Project

General

Profile

Bug #54167

segfault in radosgw-admin bucket rewrite --bucket

Added by Matthew Darwin about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

using debian package `16.2.7-1~bpo11+1` on bullseye

The following always crashes. It happens on multiple buckets.

radosgw-admin bucket rewrite --bucket <nameofbucket>

-33> 2022-02-06T15:34:07.236+0000 7f6a073f4700 10 monclient: get_auth_request con 0x7f69fc09ab10 auth_method 0
-32> 2022-02-06T15:34:07.320+0000 7f6a06bf3700 10 monclient: get_auth_request con 0x7f69f808e840 auth_method 0
-31> 2022-02-06T15:34:07.368+0000 7f6a07bf5700 10 monclient: get_auth_request con 0x7f69fc07f560 auth_method 0
-30> 2022-02-06T15:34:07.368+0000 7f6a073f4700 10 monclient: get_auth_request con 0x7f69fc09a460 auth_method 0
-29> 2022-02-06T15:34:07.532+0000 7f6a06bf3700 10 monclient: get_auth_request con 0x7f69fc08f430 auth_method 0
-28> 2022-02-06T15:34:07.628+0000 7f69dd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start
-27> 2022-02-06T15:34:07.680+0000 7f6a07bf5700 10 monclient: get_auth_request con 0x7f69fc09c9f0 auth_method 0
-26> 2022-02-06T15:34:07.748+0000 7f6a073f4700 10 monclient: get_auth_request con 0x7f69f80775e0 auth_method 0
-25> 2022-02-06T15:34:07.748+0000 7f6a06bf3700 10 monclient: get_auth_request con 0x563843258710 auth_method 0
-24> 2022-02-06T15:34:07.808+0000 7f69f6ffd700 10 monclient: tick
-23> 2022-02-06T15:34:07.808+0000 7f6955ffb700 10 monclient: tick
-22> 2022-02-06T15:34:07.896+0000 7f6a07bf5700 10 monclient: get_auth_request con 0x5638429c08a0 auth_method 0
-21> 2022-02-06T15:34:07.896+0000 7f6a073f4700 10 monclient: get_auth_request con 0x5638432a32a0 auth_method 0
-20> 2022-02-06T15:34:08.040+0000 7f6a06bf3700 10 monclient: get_auth_request con 0x563842db76b0 auth_method 0
-19> 2022-02-06T15:34:08.144+0000 7f6a07bf5700 10 monclient: get_auth_request con 0x5638437fffd0 auth_method 0
-18> 2022-02-06T15:34:08.144+0000 7f6a073f4700 10 monclient: get_auth_request con 0x5638428067f0 auth_method 0
-17> 2022-02-06T15:34:08.224+0000 7f6a06bf3700 10 monclient: get_auth_request con 0x56384370dc40 auth_method 0
-16> 2022-02-06T15:34:08.640+0000 7f6a097c26c0 -1 ** Caught signal (Segmentation fault) *
in thread 7f6a097c26c0 thread_name:radosgw-admin
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f6a0c30f140]
2: (RGWGC::send_chain(cls_rgw_obj_chain&, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&)+0xc4) [0x563840f82824]
3: (RGWRados::Object::complete_atomic_modification(DoutPrefixProvider const*)+0xc9) [0x563840cd6089]
4: (RGWRados::Object::Write::_do_write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, ceph::buffer::v15_2_0::list, std::less&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > >, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, optional_yield)+0x9c8) [0x563840ce6ab8]
5: (RGWRados::Object::Write::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, ceph::buffer::v15_2_0::list, std::less&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > >, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const, ceph::buffer::v15_2_0::list> > >&, optional_yield)+0x4c0) [0x563840ce7d60]
6: (rgw::sal::RGWRadosObject::RadosWriteOp::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, optional_yield)+0x1b) [0x563840db023b]
7: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, std::map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, ceph::buffer::v15_2_0::list, std::less&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > >, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, char const, char const*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const*, rgw_zone_set*, bool*, optional_yield)+0x2f8) [0x563840ca1b78]
8: (RGWRados::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw_placement_rule const&, RGWRados::Object::Read&, long, rgw::sal::RGWObject*, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, std::map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, ceph::buffer::v15_2_0::list, std::less&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > >, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const, ceph::buffer::v15_2_0::list> > >&, unsigned long, std::chrono::time_point&lt;ceph::real_clock, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; > >, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, DoutPrefixProvider const*, optional_yield)+0x629) [0x563840ced2a9]
9: (rgw::sal::RGWRadosObject::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw::sal::RGWObject*, unsigned short, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, DoutPrefixProvider const, optional_yield)+0x4ee) [0x563840db895e]
10: (RGWRados::rewrite_obj(RGWBucketInfo&, rgw::sal::RGWObject*, DoutPrefixProvider const*, optional_yield)+0x518) [0x563840cbcbb8]
11: main()
12: __libc_start_main()
13: _start()
NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.
15> 2022-02-06T15:34:08.808+0000 7f69f6ffd700 10 monclient: tick
-14> 2022-02-06T15:34:08.808+0000 7f6955ffb700 10 monclient: tick
-13> 2022-02-06T15:34:09.808+0000 7f6955ffb700 10 monclient: tick
-12> 2022-02-06T15:34:09.808+0000 7f69f6ffd700 10 monclient: tick
-11> 2022-02-06T15:34:10.808+0000 7f6955ffb700 10 monclient: tick
-10> 2022-02-06T15:34:10.808+0000 7f69f6ffd700 10 monclient: tick
-9> 2022-02-06T15:34:11.808+0000 7f6955ffb700 10 monclient: tick
-8> 2022-02-06T15:34:11.808+0000 7f69f6ffd700 10 monclient: tick
-7> 2022-02-06T15:34:12.808+0000 7f6955ffb700 10 monclient: tick
-6> 2022-02-06T15:34:12.808+0000 7f69f6ffd700 10 monclient: tick
-5> 2022-02-06T15:34:13.808+0000 7f6955ffb700 10 monclient: tick
-4> 2022-02-06T15:34:13.808+0000 7f69f6ffd700 10 monclient: tick
-3> 2022-02-06T15:34:14.808+0000 7f6955ffb700 10 monclient: tick
-2> 2022-02-06T15:34:14.808+0000 7f69f6ffd700 10 monclient: tick
-1> 2022-02-06T15:34:15.808+0000 7f69f6ffd700 10 monclient: tick
0> 2022-02-06T15:34:15.808+0000 7f6955ffb700 10 monclient: tick
--
logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
2/-2 (syslog threshold)
99/99 (stderr threshold)
--
pthread ID / name mapping for recent threads ---
140090391115520 / safe_timer
140092664325888 / rgw_dt_lg_renew
140093092255488 / safe_timer
140093356455680 / msgr-worker-2
140093364848384 / msgr-worker-1
140093373241088 / msgr-worker-0
140093402392256 / radosgw-admin
max_recent 500
max_new 500
log_file /var/lib/ceph/crash/2022-02-06T15:34:08.611736Z_52ee1e0f-12e9-4af5-af3d-d7521ff0ed8b/log
--- end dump of recent events ---

Related issues

Related to rgw - Bug #54433: add OPT_DATA_SYNC_RUN to gc_ops_list to initialize gc that prevents send_chain from crashing. Resolved
Related to rgw - Bug #54742: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Resolved

History

#1 Updated by Casey Bodley about 2 years ago

  • Assignee set to Daniel Gryniewicz

#2 Updated by Casey Bodley about 2 years ago

  • Related to Bug #54433: add OPT_DATA_SYNC_RUN to gc_ops_list to initialize gc that prevents send_chain from crashing. added

#3 Updated by Casey Bodley about 2 years ago

  • Status changed from New to Triaged

crash in RGWGC::send_chain() because GC isn't initalized for 'radosgw-admin bucket rewrite' or 'object rewrite' commands

#4 Updated by Casey Bodley almost 2 years ago

  • Related to Bug #54742: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#5 Updated by Casey Bodley almost 2 years ago

  • Status changed from Triaged to Resolved

Also available in: Atom PDF