Actions
Bug #47453
openchecksum failures lead to assert on OSD shutdown in lab tests
Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-09-14T11:50:01.150 INFO:tasks.ceph.osd.0.smithi186.stderr:2020-09-14T11:50:01.151+0000 7f68f74b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 13568) UID: 0
2020-09-14T11:50:01.161 INFO:teuthology.orchestra.run.smithi186.stderr:/build/ceph-16.0.0-5071-gd026253/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7fd8452fdb80 time 2020-09-14T11:50:01.157742+0000
2020-09-14T11:50:01.161 INFO:teuthology.orchestra.run.smithi186.stderr:/build/ceph-16.0.0-5071-gd026253/src/kv/RocksDBStore.cc: 1616: ceph_abort_msg("block checksum mismatch: expected 4204593633, got 976185286 in db/000073.sst offset 155593 size 4231")
2020-09-14T11:50:01.161 INFO:teuthology.orchestra.run.smithi186.stderr: ceph version 16.0.0-5071-gd026253 (d02625331c4e06ca213d9720d98137d83a87cb90) pacific (dev)
2020-09-14T11:50:01.161 INFO:teuthology.orchestra.run.smithi186.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7fd83b35ba27]
2020-09-14T11:50:01.161 INFO:teuthology.orchestra.run.smithi186.stderr: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3ec) [0x563d20f485ec]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 3: (()+0xdd3fb9) [0x563d20da7fb9]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 4: (()+0xdbf8c1) [0x563d20d938c1]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 5: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x27b) [0x563d20daeaeb]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 6: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char
> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x36b) [0x563d20e07ceb]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 7: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x18a2) [0x563d20e0bac2]
2020-09-14T11:50:01.162 INFO:teuthology.orchestra.run.smithi186.stderr: 8: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1626) [0x563d20e0ffc6]
2020-09-14T11:50:01.163 INFO:teuthology.orchestra.run.smithi186.stderr: 9: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c2) [0x563d20e29252]
2020-09-14T11:50:01.163 INFO:teuthology.orchestra.run.smithi186.stderr: 10: (BlueStore::_mount(bool, bool)+0x504) [0x563d20e29da4]
2020-09-14T11:50:01.163 INFO:teuthology.orchestra.run.smithi186.stderr: 11: (main()+0x2cb1) [0x563d2084c231]
2020-09-14T11:50:01.163 INFO:teuthology.orchestra.run.smithi186.stderr: 12: (__libc_start_main()+0xe7) [0x7fd839af6b97]
2020-09-14T11:50:01.163 INFO:teuthology.orchestra.run.smithi186.stderr: 13: (_start()+0x2a) [0x563d2086109a]
Showed up twice on two different machines:
https://pulpito.ceph.com/gregf-2020-09-14_05:25:36-rados-wip-stretch-mode-distro-basic-smithi/5433145
https://pulpito.ceph.com/gregf-2020-09-14_05:25:36-rados-wip-stretch-mode-distro-basic-smithi/5433164
Actions