Bug #50788: crash in BlueStore::Onode::put() - bluestore - Ceph

Actions

Copy link

Bug #50788

closed

crash in BlueStore::Onode::put()

Added by Neha Ojha almost 3 years ago. Updated over 2 years ago.

Status:

Duplicate

Priority:

Urgent

Assignee:

Adam Kupczyk

Target version:

% Done:

Source:

Tags:

Backport:

pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2021-05-11T20:35:02.942 INFO:tasks.ceph.osd.0.smithi001.stderr:2021-05-11T20:35:02.935+0000 7fb03deef700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  (PID: 35512) UID: 0
2021-05-11T20:35:03.016 INFO:tasks.ceph.osd.4.smithi160.stderr:ceph-osd: ../nptl/pthread_mutex_lock.c:433: __pthread_mutex_lock_full: Assertion `INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust' failed.
2021-05-11T20:35:03.017 INFO:tasks.ceph.osd.4.smithi160.stderr:*** Caught signal (Aborted) **
2021-05-11T20:35:03.017 INFO:tasks.ceph.osd.4.smithi160.stderr: in thread 7f11ddf86700 thread_name:bstore_kv_final
2021-05-11T20:35:03.019 INFO:tasks.ceph.osd.4.smithi160.stderr: ceph version 16.2.3-90-g02a5217e (02a5217eb66b90c2f3f4501df2ff1cf5a7cd7954) pacific (stable)
2021-05-11T20:35:03.019 INFO:tasks.ceph.osd.4.smithi160.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f11ee035b20]
2021-05-11T20:35:03.020 INFO:tasks.ceph.osd.4.smithi160.stderr: 2: gsignal()
2021-05-11T20:35:03.020 INFO:tasks.ceph.osd.4.smithi160.stderr: 3: abort()
2021-05-11T20:35:03.020 INFO:tasks.ceph.osd.4.smithi160.stderr: 4: /lib64/libc.so.6(+0x21b09) [0x7f11ecc87b09]
2021-05-11T20:35:03.020 INFO:tasks.ceph.osd.4.smithi160.stderr: 5: /lib64/libc.so.6(+0x2fde6) [0x7f11ecc95de6]
2021-05-11T20:35:03.021 INFO:tasks.ceph.osd.4.smithi160.stderr: 6: /lib64/libpthread.so.0(+0xa916) [0x7f11ee02d916]
2021-05-11T20:35:03.021 INFO:tasks.ceph.osd.4.smithi160.stderr: 7: (BlueStore::Onode::put()+0x29f) [0x55fc042896ef]
2021-05-11T20:35:03.021 INFO:tasks.ceph.osd.4.smithi160.stderr: 8: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x31) [0x55fc04338ec1]
2021-05-11T20:35:03.021 INFO:tasks.ceph.osd.4.smithi160.stderr: 9: (BlueStore::TransContext::~TransContext()+0x122) [0x55fc043391e2]
2021-05-11T20:35:03.021 INFO:tasks.ceph.osd.4.smithi160.stderr: 10: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x23e) [0x55fc042dc41e]
2021-05-11T20:35:03.022 INFO:tasks.ceph.osd.4.smithi160.stderr: 11: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x257) [0x55fc042ddc97]
2021-05-11T20:35:03.022 INFO:tasks.ceph.osd.4.smithi160.stderr: 12: (BlueStore::_kv_finalize_thread()+0x611) [0x55fc042eb0e1]
2021-05-11T20:35:03.022 INFO:tasks.ceph.osd.4.smithi160.stderr: 13: (BlueStore::KVFinalizeThread::entry()+0x11) [0x55fc0433e6f1]
2021-05-11T20:35:03.022 INFO:tasks.ceph.osd.4.smithi160.stderr: 14: /lib64/libpthread.so.0(+0x814a) [0x7f11ee02b14a]
2021-05-11T20:35:03.023 INFO:tasks.ceph.osd.4.smithi160.stderr: 15: clone()

/a/yuriw-2021-05-11_19:33:39-rados-wip-yuri2-testing-2021-05-11-1032-pacific-distro-basic-smithi/6109851

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Neha Ojha almost 3 years ago

Similar?

2021-06-04T05:03:40.985 INFO:tasks.ceph.osd.7.smithi140.stderr:*** Caught signal (Segmentation fault) **
2021-06-04T05:03:40.985 INFO:tasks.ceph.osd.7.smithi140.stderr: in thread 7f656d034700 thread_name:bstore_kv_final
2021-06-04T05:03:40.987 INFO:tasks.ceph.osd.7.smithi140.stderr: ceph version 16.2.4-302-gfcfd3db6 (fcfd3db6861e32b53398eb7f416bea0d37e62958) pacific (stable)
2021-06-04T05:03:40.987 INFO:tasks.ceph.osd.7.smithi140.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f65808bdb20]
2021-06-04T05:03:40.987 INFO:tasks.ceph.osd.7.smithi140.stderr: 2: (BlueStore::Onode::put()+0x2ab) [0x5622c4be2f9b]
2021-06-04T05:03:40.988 INFO:tasks.ceph.osd.7.smithi140.stderr: 3: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x31) [0x5622c4c92761]
2021-06-04T05:03:40.988 INFO:tasks.ceph.osd.7.smithi140.stderr: 4: (BlueStore::TransContext::~TransContext()+0x122) [0x5622c4c92a82]
2021-06-04T05:03:40.988 INFO:tasks.ceph.osd.7.smithi140.stderr: 5: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x23e) [0x5622c4c35cbe]
2021-06-04T05:03:40.988 INFO:tasks.ceph.osd.7.smithi140.stderr: 6: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x257) [0x5622c4c37537]
2021-06-04T05:03:40.988 INFO:tasks.ceph.osd.7.smithi140.stderr: 7: (BlueStore::_kv_finalize_thread()+0x54e) [0x5622c4c448be]
2021-06-04T05:03:40.989 INFO:tasks.ceph.osd.7.smithi140.stderr: 8: (BlueStore::KVFinalizeThread::entry()+0x11) [0x5622c4c97f91]
2021-06-04T05:03:40.989 INFO:tasks.ceph.osd.7.smithi140.stderr: 9: /lib64/libpthread.so.0(+0x814a) [0x7f65808b314a]
2021-06-04T05:03:40.989 INFO:tasks.ceph.osd.7.smithi140.stderr: 10: clone()

/a/yuriw-2021-06-02_18:33:05-rados-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147437 - no logs

Actions

Copy link

Updated by Neha Ojha over 2 years ago

2021-08-06T18:34:07.077 INFO:tasks.ceph.osd.5.smithi094.stderr:*** Caught signal (Segmentation fault) **
2021-08-06T18:34:07.077 INFO:tasks.ceph.osd.5.smithi094.stderr: in thread 7faad6d8b700 thread_name:bstore_kv_final
2021-08-06T18:34:07.082 INFO:tasks.ceph.osd.5.smithi094.stderr: ceph version 17.0.0-6744-g3c0f8c81 (3c0f8c8164075af7aac4d1f2805d3f4580709461) quincy (dev)
2021-08-06T18:34:07.082 INFO:tasks.ceph.osd.5.smithi094.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7faae9414b20]
2021-08-06T18:34:07.082 INFO:tasks.ceph.osd.5.smithi094.stderr: 2: __pthread_mutex_lock()
2021-08-06T18:34:07.083 INFO:tasks.ceph.osd.5.smithi094.stderr: 3: (BlueStore::Onode::put()+0xa4) [0x55efed6fdb98]
2021-08-06T18:34:07.083 INFO:tasks.ceph.osd.5.smithi094.stderr: 4: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x15) [0x55efed7be7cf]
2021-08-06T18:34:07.084 INFO:tasks.ceph.osd.5.smithi094.stderr: 5: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x31) [0x55efed7be7eb]
2021-08-06T18:34:07.085 INFO:tasks.ceph.osd.5.smithi094.stderr: 6: (BlueStore::TransContext::~TransContext()+0x12a) [0x55efed7be92a]
2021-08-06T18:34:07.085 INFO:tasks.ceph.osd.5.smithi094.stderr: 7: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x9b1) [0x55efed76cba3]
2021-08-06T18:34:07.085 INFO:tasks.ceph.osd.5.smithi094.stderr: 8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xcf1) [0x55efed76dcf9]
2021-08-06T18:34:07.086 INFO:tasks.ceph.osd.5.smithi094.stderr: 9: (BlueStore::_kv_finalize_thread()+0x1025) [0x55efed77173d]
2021-08-06T18:34:07.086 INFO:tasks.ceph.osd.5.smithi094.stderr: 10: (BlueStore::KVFinalizeThread::entry()+0x11) [0x55efed7c470b]
2021-08-06T18:34:07.086 INFO:tasks.ceph.osd.5.smithi094.stderr: 11: (Thread::entry_wrapper()+0x43) [0x55efed8eb83d]
2021-08-06T18:34:07.086 INFO:tasks.ceph.osd.5.smithi094.stderr: 12: (Thread::_entry_func(void*)+0xd) [0x55efed8eb859]
2021-08-06T18:34:07.087 INFO:tasks.ceph.osd.5.smithi094.stderr: 13: /lib64/libpthread.so.0(+0x814a) [0x7faae940a14a]
2021-08-06T18:34:07.087 INFO:tasks.ceph.osd.5.smithi094.stderr: 14: clone()

/a/yuriw-2021-08-06_16:31:19-rados-wip-yuri-master-8.6.21-distro-basic-smithi/6324555 - no logs

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Subject changed from __pthread_mutex_lock_full: Assertion `INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust' failed to crash in BlueStore::Onode::put()
Priority changed from Normal to High

2021-08-24T04:12:52.879 INFO:tasks.ceph.osd.7.smithi124.stderr:*** Caught signal (Segmentation fault) **
2021-08-24T04:12:52.879 INFO:tasks.ceph.osd.7.smithi124.stderr: in thread 7f03c9579700 thread_name:bstore_kv_final
2021-08-24T04:12:52.882 INFO:tasks.ceph.osd.7.smithi124.stderr: ceph version 16.2.5-434-g20e257e1 (20e257e138851c8cfc5bbfcef07d1f8e68373538) pacific (stable)
2021-08-24T04:12:52.882 INFO:tasks.ceph.osd.7.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f03d8e2db20]
2021-08-24T04:12:52.882 INFO:tasks.ceph.osd.7.smithi124.stderr: 2: __pthread_mutex_lock()
2021-08-24T04:12:52.883 INFO:tasks.ceph.osd.7.smithi124.stderr: 3: (BlueStore::Onode::put()+0x2bf) [0x55be01c046ff]
2021-08-24T04:12:52.883 INFO:tasks.ceph.osd.7.smithi124.stderr: 4: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x31) [0x55be01cb44a1]
2021-08-24T04:12:52.883 INFO:tasks.ceph.osd.7.smithi124.stderr: 5: (BlueStore::TransContext::~TransContext()+0x122) [0x55be01cb47c2]
2021-08-24T04:12:52.883 INFO:tasks.ceph.osd.7.smithi124.stderr: 6: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x23e) [0x55be01c5756e]
2021-08-24T04:12:52.883 INFO:tasks.ceph.osd.7.smithi124.stderr: 7: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x257) [0x55be01c58de7]
2021-08-24T04:12:52.884 INFO:tasks.ceph.osd.7.smithi124.stderr: 8: (BlueStore::_kv_finalize_thread()+0x611) [0x55be01c66361]
2021-08-24T04:12:52.884 INFO:tasks.ceph.osd.7.smithi124.stderr: 9: (BlueStore::KVFinalizeThread::entry()+0x11) [0x55be01cb9d31]
2021-08-24T04:12:52.884 INFO:tasks.ceph.osd.7.smithi124.stderr: 10: /lib64/libpthread.so.0(+0x814a) [0x7f03d8e2314a]
2021-08-24T04:12:52.884 INFO:tasks.ceph.osd.7.smithi124.stderr: 11: clone()

Logs and coredump present this time yuriw-2021-08-23_19:24:05-rados-wip-yuri4-testing-2021-08-23-0812-pacific-distro-basic-smithi/6353843/remote/smithi124/coredump

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Assignee set to Adam Kupczyk

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #51960: octopus: Assertion `new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio) added

Actions

Copy link

Updated by Neha Ojha over 2 years ago

2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6f9e8 in 0x56378da5be10) _finish_write discard buffer(0x5637a08c2600 space 0x56379ec6f9e8 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6fac8 in 0x56378da5be10) _finish_write discard buffer(0x5637a08c34a0 space 0x56379ec6fac8 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6fba8 in 0x56378da5be10) _finish_write discard buffer(0x56379f9e7ce0 space 0x56379ec6fba8 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6fc88 in 0x56378da5be10) _finish_write discard buffer(0x5637a3058540 space 0x56379ec6fc88 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6fd68 in 0x56378da5be10) _finish_write discard buffer(0x5637a283d0e0 space 0x56379ec6fd68 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6fe48 in 0x56378da5be10) _finish_write discard buffer(0x563797275980 space 0x56379ec6fe48 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore.BufferSpace(0x56379ec6ff28 in 0x56378da5be10) _finish_write discard buffer(0x5637967d2f00 space 0x56379ec6ff28 0x0~2000 writing nocache)
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish  txc 0x563795e5e900 done
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish  txc 0x56378ef0e900 done
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish  txc 0x5637985fa900 done
2021-09-20T06:01:01.997+0000 7fa90964d700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish osr 0x56379fac54a0 q now empty
2021-09-20T06:01:01.997+0000 7fa90964d700 10 bluestore(/var/lib/ceph/osd/ceph-0) _txc_release_alloc(sync) 0x563795e5e900 []
...
2021-09-20T06:01:02.016+0000 7fa90964d700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fa90964d700 thread_name:bstore_kv_final

 ceph version 17.0.0-7814-gae31c79b (ae31c79bc8ac09a6d68edebf7ba88cee96c659f3) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7fa91ecf7b20]
 2: (BlueStore::Onode::put()+0x2dc) [0x56378a4cfefc]
 3: (std::_Rb_tree<boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive_ptr<BlueStore::Onode>, std::_Identity<boost::intrusive_ptr<BlueStore::Onode> >, std::less<boost::intrusive_ptr<BlueStore::Onode> >, std::allocator<boost::intrusive_ptr<BlueStore::Onode> > >::_M_erase(std::_Rb_tree_node<boost::intrusive_ptr<BlueStore::Onode> >*)+0x31) [0x56378a5939f1]
 4: (BlueStore::TransContext::~TransContext()+0x147) [0x56378a593b67]
 5: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x24e) [0x56378a52707e]
 6: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x227) [0x56378a52a817]
 7: (BlueStore::_kv_finalize_thread()+0x611) [0x56378a5610b1]
 8: (BlueStore::KVFinalizeThread::entry()+0x11) [0x56378a59bb51]
 9: /lib64/libpthread.so.0(+0x814a) [0x7fa91eced14a]
 10: clone()

/a/sage-2021-09-20_04:54:37-rados-wip-sage-testing-2021-09-19-2154-distro-basic-smithi/6398464/remote/smithi158/log/ceph-osd.0.log.gz

Actions

Copy link