Project

General

Profile

Actions

Bug #54119

closed

bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failed

Added by Mykola Golub about 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

100%

Source:
Tags:
backport_processed
Backport:
quincy,pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

   -16> 2022-02-02T10:42:54.605+0000 7faf3a7fc700 10 trim: trimming bilog shard 0 of :test_bucket2[]) at marker 99999999
   -15> 2022-02-02T10:42:54.605+0000 7faf3a7fc700  0 ERROR: empty bucket_id for bucket operation
   -14> 2022-02-02T10:42:54.605+0000 7faf3a7fc700 20 open_bucket_index_shard: open_bucket_index_pool() returned -5
   -13> 2022-02-02T10:42:54.605+0000 7faf3a7fc700  0 ERROR: open_bucket_index_shard() returned ret=-5
   -12> 2022-02-02T10:42:54.605+0000 7faf3a7fc700 20 run: stack=0x7faf280c6840 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
   -11> 2022-02-02T10:42:54.605+0000 7faf3a7fc700 20 cr:s=0x7faf2802cec0:op=0x7faf2806b180:19RGWRadosBILogTrimCR: operate()
   -10> 2022-02-02T10:42:54.605+0000 7faf3a7fc700 20 cr:s=0x7faf2802cec0:op=0x7faf2806b180:19RGWRadosBILogTrimCR: operate()
    -9> 2022-02-02T10:42:54.605+0000 7fafdf7e6700 20 rados_obj.operate() r=-2 bl.length=0
    -8> 2022-02-02T10:42:54.605+0000 7fafdf7e6700 10 cache put: name=default.rgw.log++bucket.sync-target-hints.airport-forecast-charts info.flags=0x0
    -7> 2022-02-02T10:42:54.605+0000 7fafdf7e6700 10 adding default.rgw.log++bucket.sync-target-hints.airport-forecast-charts to cache LRU end
    -6> 2022-02-02T10:42:54.605+0000 7fafdf7e6700 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.airport-forecast-charts:fce69a1c-c3c0-46ea-bad1-25175715bbe5.46393607.4
    -5> 2022-02-02T10:42:54.609+0000 7faf3a7fc700 -1 *** Caught signal (Segmentation fault) **
 in thread 7faf3a7fc700 thread_name:sync-log-trim

 ceph version 15.2.15-83-gf72054fa653 (f72054fa65354688da034079d54554bf34373d94) octopus (stable)
 1: (()+0x393e0) [0x7fb0241873e0]
 2: (()+0x8cc7d) [0x7fb023bfec7d]
 3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x70) [0x7fb023bd6050]
 4: (RGWRadosBILogTrimCR::send_request()+0x1c3) [0x7fb024c90b93]
 5: (RGWSimpleCoroutine::state_send_request()+0xe) [0x7fb024c884be]
 6: (RGWSimpleCoroutine::operate()+0x8c) [0x7fb024c8dd9c]
 7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x63) [0x7fb024c8b8f3]
 8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x2b5) [0x7fb024c8c6d5]
 9: (RGWSyncLogTrimThread::process()+0x1fc) [0x7fb024d58eac]
 10: (RGWRadosThread::Worker::entry()+0x176) [0x7fb024d1f0e6]
 11: (()+0x8539) [0x7fb01a02e539]
 12: (clone()+0x3f) [0x7fb024249e0f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

In RGWRadosBILogTrimCR we don't check the bucket shard init (bs.init()) return code, and proceed with send_request even if the init failed, crashing when trying to use uninitialized bs.bucket_obj.


Related issues 4 (0 open4 closed)

Has duplicate rgw - Bug #56444: ceph-rgw daemon crashes with Caught signal (Segmentation fault) in thread 7f2f95529700 thread_name:sync-log-trimDuplicate

Actions
Copied to rgw - Backport #54143: quincy: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failedResolvedMykola GolubActions
Copied to rgw - Backport #54144: pacific: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failedResolvedMykola GolubActions
Copied to rgw - Backport #54145: octopus: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failedRejectedMykola GolubActions
Actions #1

Updated by Mykola Golub about 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 44892
Actions #2

Updated by Casey Bodley about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #3

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54143: quincy: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failed added
Actions #4

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54144: pacific: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failed added
Actions #5

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54145: octopus: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket shard init failed added
Actions #6

Updated by Casey Bodley almost 2 years ago

  • Has duplicate Bug #56444: ceph-rgw daemon crashes with Caught signal (Segmentation fault) in thread 7f2f95529700 thread_name:sync-log-trim added
Actions #7

Updated by Rishat Azizov almost 2 years ago

Hello!

Could you please backport this to octopus?

Thanks.

Actions #8

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions #9

Updated by Konstantin Shalygin about 1 year ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF