Bug #14533
Updated by David Zafman over 8 years ago
teuthology.ovh.sepia.ceph.com:/a/teuthology-2016-01-27_16:15:46-upgrade:hammer-x-wip-diag-14438-distro-basic-openstack/12016 teuthology didn't gather up information from the test nodes as of this moment, but I was able to connect briefly to the machines and got this information: wip-diag-14438 is equivalent to jewel. OSD still running hammer. ubuntu@target077206:/var/lib/ceph/osd/ceph-5/current$ ls -R 3.0s1_head 3.0s1_head: temp\u3.0s0\u0\u34165\u3__head_00000000__fffffffffffffffb_ffffffffffffffff_1 Object named: temp_3.0s0_0_34165_3 <pre> 2016-01-27 17:02:12.328833 7f06ab5978c0 0 ceph version 0.94.5-227-gd54840b (d54840bf4a70fc65285bbfdff0c7bf8f579643b1), process ceph-osd, pid 12493 ... 2016-01-27 17:24:58.024848 7f069df65700 15 filestore(/var/lib/ceph/osd/ceph-5) write 3.0s1_head/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 0~4194304 2016-01-27 17:24:58.029039 7f069df65700 10 filestore(/var/lib/ceph/osd/ceph-5) write 3.0s1_head/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 0~4194304 = 4194304 2016-01-27 17:24:58.029091 7f069df65700 15 filestore(/var/lib/ceph/osd/ceph-5) setattrs 3.0s1_head/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 2016-01-27 17:24:58.029124 7f069df65700 10 filestore oid: 0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 not skipping op, *spos 34858.0.19 2016-01-27 17:24:58.029150 7f069df65700 10 filestore(/var/lib/ceph/osd/ceph-5) setattrs 3.0s1_head/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 = 0 2016-01-27 17:24:59.621712 7f06915ed700 10 osd.5 pg_epoch: 840 pg[3.0s1( v 828'531 lc 763'494 (0'0,828'531] local-les=835 n=6 ec=30 les/c 835/691 840/840/683) [1,5,4] r=1 lpr=840 pi=683-839/2 crt=828'531 inactive m=7] on_change_cleanup: Removing oid 0/temp_3.0s0_0_34165_3/head//-5 from the temp collection 2016-01-27 17:24:59.629366 7f069e766700 15 filestore(/var/lib/ceph/osd/ceph-5) remove 3.0s1_TEMP/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 2016-01-27 17:24:59.629415 7f069e766700 10 filestore(/var/lib/ceph/osd/ceph-5) remove 3.0s1_TEMP/0/temp_3.0s0_0_34165_3/head//-5/18446744073709551615/1 = -2 .... 2016-01-27 17:25:25.103061 7f069e766700 5 filestore(/var/lib/ceph/osd/ceph-5) _do_op 0x9026cd0 seq 35628 osr(3.0s1 0x83f2e00)/0x83f2e00 start 2016-01-27 17:25:25.103064 7f069e766700 10 filestore(/var/lib/ceph/osd/ceph-5) _do_transaction on 0xb748140 2016-01-27 17:25:25.103068 7f069e766700 15 filestore(/var/lib/ceph/osd/ceph-5) remove 3.0s1_head/0//head//3/18446744073709551615/1 2016-01-27 17:25:25.103086 7f069e766700 20 filestore(/var/lib/ceph/osd/ceph-5) lfn_unlink: clearing omap on 0//head//3/18446744073709551615/1 in cid 3.0s1_head 2016-01-27 17:25:25.103093 7f069e766700 10 filestore oid: 0//head//3/18446744073709551615/1 not skipping op, *spos 35628.0.0 2016-01-27 17:25:25.103095 7f069e766700 10 filestore > header.spos 0.0.0 2016-01-27 17:25:25.103097 7f069e766700 20 filestore remove_map_header: removing 5711 oid 0//head//3/18446744073709551615/1 2016-01-27 17:25:25.103103 7f069e766700 20 filestore clear_header: clearing seq 5711 2016-01-27 17:25:25.104726 7f069e766700 10 filestore(/var/lib/ceph/osd/ceph-5) remove 3.0s1_head/0//head//3/18446744073709551615/1 = 0 2016-01-27 17:25:25.105583 7f069e766700 0 filestore(/var/lib/ceph/osd/ceph-5) error (39) Directory not empty not handled on operation 0x91627a8 (35628.0.1, or op 1, counting from 0) 2016-01-27 17:25:25.105606 7f069e766700 0 filestore(/var/lib/ceph/osd/ceph-5) ENOTEMPTY suggests garbage data in osd data dir 2016-01-27 17:25:25.105609 7f069e766700 0 filestore(/var/lib/ceph/osd/ceph-5) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.0s1_head", "oid": "0\/\/head\/\/3\/18446744073709551615\/1" }, { "op_num": 1, "op_name": "rmcoll", "collection": "3.0s1_head" }, { "op_num": 2, "op_name": "rmcoll", "collection": "3.0s1_TEMP" } ] } 2016-01-27 17:25:25.113874 7f069e766700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f069e766700 time 2016-01-27 17:25:25.105731 os/FileStore.cc: 2761: FAILED assert(0 == "unexpected error") </pre>