Project

General

Profile

Bug #23492

Updated by David Zafman almost 6 years ago


dzafman-2018-03-28_15:20:23-rados:standalone-wip-zafman-testing-distro-basic-smithi/2331804

In TEST_rados_get_bad_size_shard_0 a ceph-osd startup of osd.3 resulted in the following crash. This was incidental to taking osd.3 down to corrupt an object using ceph-objectstore-tool.

<pre>

2018-03-28T22:47:50.571 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 42 2674 bytes
2018-03-28T22:47:50.571 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 41 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 42 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 42 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 43 - loading and decoding 0x556d1fae3b00
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 43 2674 bytes
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 42 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 43 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 43 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 44 - loading and decoding 0x556d1fae3f80
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 44 2674 bytes
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 43 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.572 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 44 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.573 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 44 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.573 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 45 - loading and decoding 0x556d1fae4400
2018-03-28T22:47:50.573 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 45 2674 bytes
2018-03-28T22:47:50.573 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 44 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 45 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 45 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 46 - loading and decoding 0x556d1fae4880
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 46 2674 bytes
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 45 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 46 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 46 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 47 - loading and decoding 0x556d1fae4d00
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 47 2674 bytes
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 46 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] handle_advance_map [3,0,2]/[3,0,2] -- 3/3
2018-03-28T22:47:50.574 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 47 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] state<Reset>: Reset advmap
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 pg_epoch: 47 pg[1.1( empty local-lis/les=38/39 n=0 ec=2/2 lis/c 38/38 les/c/f 39/39/0 38/38/38) [3,0,2] r=0 lpr=39 crt=0'0 mlcod 0'0 unknown mbc={}] check_recovery_sources no source osds () went down
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 20 osd.3 52 get_map 48 - loading and decoding 0x556d1fae5180
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.023 7f35e22e2700 10 osd.3 52 add_map_bl 48 2674 bytes
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout:2018-03-28 22:40:05.027 7f35e22e2700 -1 *** Caught signal (Aborted) **
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout: in thread 7f35e22e2700 thread_name:tp_osd_tp
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout:
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout: ceph version 13.0.1-3362-gab967e9 (ab967e9551853fdb3fcd2ddb071a2bb55e80e6d1) mimic (dev)
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout: 1: (()+0x8fef70) [0x556d1d380f70]
2018-03-28T22:47:50.575 INFO:tasks.workunit.client.0.smithi132.stdout: 2: (()+0x11390) [0x7f3600f1a390]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 3: (gsignal()+0x38) [0x7f3600667428]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 4: (abort()+0x16a) [0x7f360066902a]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f3602938965]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f36028a0ee6]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 7: (()+0x71df31) [0x7f36028a0f31]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 8: (()+0x71d464) [0x7f36028a0464]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 9: (OSDMap::decode(ceph::buffer::list::iterator&)+0x1901) [0x7f36025cd961]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 10: (OSDMap::decode(ceph::buffer::list&)+0x31) [0x7f36025cecc1]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 11: (OSDService::try_get_map(unsigned int)+0x6bc) [0x556d1ce4b5fc]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x197) [0x556d1ce53ec7]
2018-03-28T22:47:50.576 INFO:tasks.workunit.client.0.smithi132.stdout: 13: (OSD::dequeue_peering_evt(PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xd4) [0x556d1ce54714]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 14: (PGPeeringItem::run(OSD*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x4d) [0x556d1d0afc3d]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1092) [0x556d1ce49b32]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x452) [0x7f3602457ef2]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f3602459ef0]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 18: (()+0x76ba) [0x7f3600f106ba]
2018-03-28T22:47:50.577 INFO:tasks.workunit.client.0.smithi132.stdout: 19: (clone()+0x6d) [0x7f360073941d]

</pre>

Back