Bug #56444: ceph-rgw daemon crashes with Caught signal (Segmentation fault) in thread 7f2f95529700 thread_name:sync-log-trim - rgw - Ceph

Actions

Copy link

Bug #56444

closed

ceph-rgw daemon crashes with Caught signal (Segmentation fault) in thread 7f2f95529700 thread_name:sync-log-trim

Added by Rishat Azizov almost 2 years ago. Updated almost 2 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v15.2.16

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Hello!

Periodically we are getting errors on rgw daemons and its crash. Details below:

radosgw[2572680]: *** Caught signal (Segmentation fault) **
radosgw[2572680]:  in thread 7f2f95529700 thread_name:sync-log-trim
radosgw[2572680]:  ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
radosgw[2572680]:  1: (()+0x12ce0) [0x7f2fbfb23ce0]
radosgw[2572680]:  2: (()+0x98fb1) [0x7f2fca0aafb1]
radosgw[2572680]:  3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7f2fca083164]
radosgw[2572680]:  4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7f2fcaaa7077]
radosgw[2572680]:  5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7f2fcaa9ea03]
radosgw[2572680]:  6: (RGWSimpleCoroutine::operate()+0xac) [0x7f2fcaaa3fcc]
radosgw[2572680]:  7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7f2fcaaa17f7]
radosgw[2572680]:  8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f2fcaaa2621]
radosgw[2572680]:  9: (RGWSyncLogTrimThread::process()+0x200) [0x7f2fcab693a0]
radosgw[2572680]:  10: (RGWRadosThread::Worker::entry()+0x176) [0x7f2fcab31496]
radosgw[2572680]:  11: (()+0x81cf) [0x7f2fbfb191cf]
radosgw[2572680]:  12: (clone()+0x43) [0x7f2fbe160d83]
radosgw[2572680]: 2022-07-03T16:20:11.202+0000 7f2f95529700 -1 *** Caught signal (Segmentation fault) **
radosgw[2572680]:  in thread 7f2f95529700 thread_name:sync-log-trim
radosgw[2572680]:  ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
radosgw[2572680]:  1: (()+0x12ce0) [0x7f2fbfb23ce0]
radosgw[2572680]:  2: (()+0x98fb1) [0x7f2fca0aafb1]
radosgw[2572680]:  3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7f2fca083164]
radosgw[2572680]:  4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7f2fcaaa7077]
radosgw[2572680]:  5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7f2fcaa9ea03]
radosgw[2572680]:  6: (RGWSimpleCoroutine::operate()+0xac) [0x7f2fcaaa3fcc]
radosgw[2572680]:  7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7f2fcaaa17f7]
radosgw[2572680]:  8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f2fcaaa2621]
radosgw[2572680]:  9: (RGWSyncLogTrimThread::process()+0x200) [0x7f2fcab693a0]
radosgw[2572680]:  10: (RGWRadosThread::Worker::entry()+0x176) [0x7f2fcab31496]
radosgw[2572680]:  11: (()+0x81cf) [0x7f2fbfb191cf]
radosgw[2572680]:  12: (clone()+0x43) [0x7f2fbe160d83]
radosgw[2572680]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
radosgw[2572680]:  -6876> 2022-07-03T15:45:34.129+0000 7f2ee23c3700 -1 res_query() failed
radosgw[2572680]:  -2638> 2022-07-03T16:06:56.259+0000 7f2ef53e9700 -1 res_query() failed
radosgw[2572680]:   -810> 2022-07-03T16:16:08.064+0000 7f2eeabd4700 -1 res_query() failed
radosgw[2572680]:      0> 2022-07-03T16:20:11.202+0000 7f2f95529700 -1 *** Caught signal (Segmentation fault) **
radosgw[2572680]:  in thread 7f2f95529700 thread_name:sync-log-trim
radosgw[2572680]:  ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
radosgw[2572680]:  1: (()+0x12ce0) [0x7f2fbfb23ce0]
radosgw[2572680]:  2: (()+0x98fb1) [0x7f2fca0aafb1]
radosgw[2572680]:  3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7f2fca083164]
radosgw[2572680]:  4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7f2fcaaa7077]
radosgw[2572680]:  5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7f2fcaa9ea03]
radosgw[2572680]:  6: (RGWSimpleCoroutine::operate()+0xac) [0x7f2fcaaa3fcc]
radosgw[2572680]:  7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7f2fcaaa17f7]
radosgw[2572680]:  8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f2fcaaa2621]
radosgw[2572680]:  9: (RGWSyncLogTrimThread::process()+0x200) [0x7f2fcab693a0]
radosgw[2572680]:  10: (RGWRadosThread::Worker::entry()+0x176) [0x7f2fcab31496]
radosgw[2572680]:  11: (()+0x81cf) [0x7f2fbfb191cf]
radosgw[2572680]:  12: (clone()+0x43) [0x7f2fbe160d83]
radosgw[2572680]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
radosgw[2572680]:  -6876> 2022-07-03T15:45:34.129+0000 7f2ee23c3700 -1 res_query() failed
radosgw[2572680]:  -2638> 2022-07-03T16:06:56.259+0000 7f2ef53e9700 -1 res_query() failed
radosgw[2572680]:   -810> 2022-07-03T16:16:08.064+0000 7f2eeabd4700 -1 res_query() failed
radosgw[2572680]:      0> 2022-07-03T16:20:11.202+0000 7f2f95529700 -1 *** Caught signal (Segmentation fault) **
radosgw[2572680]:  in thread 7f2f95529700 thread_name:sync-log-trim
radosgw[2572680]:  ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
radosgw[2572680]:  1: (()+0x12ce0) [0x7f2fbfb23ce0]
radosgw[2572680]:  2: (()+0x98fb1) [0x7f2fca0aafb1]
radosgw[2572680]:  3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7f2fca083164]
radosgw[2572680]:  4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7f2fcaaa7077]
radosgw[2572680]:  5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7f2fcaa9ea03]
radosgw[2572680]:  6: (RGWSimpleCoroutine::operate()+0xac) [0x7f2fcaaa3fcc]
radosgw[2572680]:  7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7f2fcaaa17f7]
radosgw[2572680]:  8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f2fcaaa2621]
radosgw[2572680]:  9: (RGWSyncLogTrimThread::process()+0x200) [0x7f2fcab693a0]
radosgw[2572680]:  10: (RGWRadosThread::Worker::entry()+0x176) [0x7f2fcab31496]
radosgw[2572680]:  11: (()+0x81cf) [0x7f2fbfb191cf]
radosgw[2572680]:  12: (clone()+0x43) [0x7f2fbe160d83]
radosgw[2572680]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
systemd[1]: ceph-radosgw@rgw.str-dc3-2-5-17.service: Main process exited, code=killed, status=11/SEGV
systemd[1]: ceph-radosgw@rgw.str-dc3-2-5-17.service: Failed with result 'signal'.
systemd[1]: ceph-radosgw@rgw.str-dc3-2-5-17.service: Service RestartSec=100ms expired, scheduling restart.
systemd[1]: ceph-radosgw@rgw.str-dc3-2-5-17.service: Scheduled restart job, restart counter is at 75.
systemd[1]: Stopped Ceph rados gateway.
systemd[1]: Started Ceph rados gateway.

Could you please help with this? Please let me know if you need additional information.

Thanks.

Related issues 1 (0 open — 1 closed)