Actions
Bug #4059
closedosd: ENOTEMPTY unhandled for remove op
Status:
Duplicate
Priority:
High
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
bobtail
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This occurred on wip_bobtail_f in a local vstart 11-osd cluster which I was trying to use to reproduce #4052 by causing some thrashing and some osds to stay down for a while while there were some rados benches and several unused pools.
2013-02-08 12:16:30.453731 7f79222ee780 10 journal op_apply_finish 1758 open_ops 1 -> 0, max_applied_seq 1757 -> 1758 2013-02-08 12:16:30.453735 7f79222ee780 3 journal journal_replay: r = 0, op_seq now 1758 2013-02-08 12:16:30.453747 7f79222ee780 2 journal read_entry 100585472 : seq 1759 263 bytes 2013-02-08 12:16:30.453750 7f79222ee780 3 journal journal_replay: applying op seq 1759 2013-02-08 12:16:30.453760 7f79222ee780 10 journal op_apply_start 1759 open_ops 0 -> 1 2013-02-08 12:16:30.453763 7f79222ee780 10 filestore(dev/osd2) _do_transaction on 0x3463a20 2013-02-08 12:16:30.453827 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1892.0.9 > current pos 1759.0.0, now or in future, SKIPPING REPLAY 2013-02-08 12:16:30.453890 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1895.0.8 > current pos 1759.0.1, now or in future, SKIPPING REPLAY 2013-02-08 12:16:30.453911 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1753.0.0 < current pos 1759.0.2, in past, will replay 2013-02-08 12:16:30.454098 7f79222ee780 0 filestore(dev/osd2) error (39) Directory not empty not handled on operation 21 (1759.0.2, or op 2, counting from 0) 2013-02-08 12:16:30.454111 7f79222ee780 0 filestore(dev/osd2) ENOTEMPTY suggests garbage data in osd data dir 2013-02-08 12:16:30.454112 7f79222ee780 0 filestore(dev/osd2) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "FORREMOVAL_15_8.21", "oid": "b0c75721\/benchmark_data_vit_17179_object70\/head\/\/8"}, { "op_num": 1, "op_name": "remove", "collection": "FORREMOVAL_15_8.21", "oid": "9327c821\/benchmark_data_vit_17179_object79\/head\/\/8"}, { "op_num": 2, "op_name": "rmcoll", "collection": "FORREMOVAL_15_8.21"}]} 2013-02-08 12:16:30.461801 7f79222ee780 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int)' thread 7f79222 ee780 time 2013-02-08 12:16:30.454158 os/FileStore.cc: 2667: FAILED assert(0 == "unexpected error") ceph version 0.56-663-g46d7dbd (46d7dbd3472f26926c6d048bfc3c150074bfd283) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xf86351] 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x3e90) [0xee971a] 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0xfc) [0xee3b56] 4: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x31) [0xf036bf] 5: (JournalingObjectStore::journal_replay(unsigned long)+0x97c) [0xf11520] 6: (FileStore::mount()+0x519f) [0xee07eb] 7: (OSD::do_convertfs(ObjectStore*)+0x32) [0xd0576e] 8: (OSD::convertfs(std::string const&, std::string const&)+0x87) [0xd067d3] 9: (main()+0x3869) [0xc2efc5] 10: (__libc_start_main()+0xfd) [0x7f792008dc4d] 11: ./ceph-osd() [0xc2b459]
Full log is attached.
Files
Actions