https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2019-03-13T16:29:38Z
Ceph
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=131952
2019-03-13T16:29:38Z
Sage Weil
sage@newdream.net
<ul></ul><p>ceph-post-file: 26dab2cb-36c9-40de-8455-1379406477e8</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=133896
2019-04-09T22:22:44Z
Sage Weil
sage@newdream.net
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Need More Info</i></li><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>High</i></li></ul><p>logging level isn't high enough to tell what data is in this pg. :(</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=135671
2019-04-29T21:02:25Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>12</i></li></ul><p>A user saw this and uploaded with debug 20 on OSD and bluestone: 2d8d22f4-580b-4b57-a13a-f49dade34ba7</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=135927
2019-05-02T16:33:43Z
Sage Weil
sage@newdream.net
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/27929">https://github.com/ceph/ceph/pull/27929</a></p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=135928
2019-05-02T16:34:00Z
Sage Weil
sage@newdream.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-10 priority-6 priority-high2 closed" href="/issues/39152">Bug #39152</a>: nautilus osd crash: Caught signal (Aborted) tp_osd_tp</i> added</li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=136085
2019-05-04T13:27:47Z
Lenz Grimmer
<ul><li><strong>Pull request ID</strong> set to <i>27929</i></li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=136252
2019-05-07T17:50:31Z
Sage Weil
sage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li><li><strong>Backport</strong> set to <i>nautilus, mimic, luminous</i></li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=136497
2019-05-10T10:59:02Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/39692">Backport #39692</a>: mimic: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)</i> added</li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=136500
2019-05-10T10:59:09Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/39693">Backport #39693</a>: nautilus: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)</i> added</li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=136502
2019-05-10T10:59:17Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-6 priority-4 priority-default closed" href="/issues/39694">Backport #39694</a>: luminous: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)</i> added</li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140011
2019-07-03T09:43:52Z
Igor Fedotov
igor.fedotov@croit.io
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/40642">Bug #40642</a>: Bluestore crash due to mass activation on another pool</i> added</li></ul>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140257
2019-07-09T16:15:22Z
Edward Kalk
<ul></ul><p>We have hit this as well, it was triggered when I rebooted a node. A few OSD on other hosts crashed. Here's some log:<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2019-07-08 16:00:54.962 7fe236221700 -1 bluestore(/var/lib/ceph/osd/ceph-12) <em>txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: /build/ceph-14.2.1/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7fe236221700 time 2019-07-08 16:00:54.965119<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: /build/ceph-14.2.1/src/os/bluestore/BlueStore.cc: 11089: abort()<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 1: (ceph::</em>_ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xda) [0x850a01]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x296a) [0xe44a2a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 3: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x5e6) [0xe48f96]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 4: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7f) [0xa0099f]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 5: (PG::_delete_some(ObjectStore::Transaction*)+0x710) [0xa64b30]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 6: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x71) [0xa658f1]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 7: (boost::statechart::simple_state<PG::RecoveryState::Deleting, PG::RecoveryState::ToDelete, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x131) [0xaaf151]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0xa82b0b]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0xa72052]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x9ac6a4]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 11: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&)+0xd2) [0x9ac982]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x9a07dd]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xfc37bc]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xfc6970]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 15: (()+0x76ba) [0x7fe2557446ba]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 16: (clone()+0x6d) [0x7fe254d4b41d]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2019-07-08 16:00:54.974 7fe236221700 -1 /build/ceph-14.2.1/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7fe236221700 time 2019-07-08 16:00:54.965119<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: /build/ceph-14.2.1/src/os/bluestore/BlueStore.cc: 11089: abort()<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xda) [0x850a01]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x296a) [0xe44a2a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 3: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x5e6) [0xe48f96]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 4: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7f) [0xa0099f]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 5: (PG::_delete_some(ObjectStore::Transaction*)+0x710) [0xa64b30]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 6: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x71) [0xa658f1]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 7: (boost::statechart::simple_state<PG::RecoveryState::Deleting, PG::RecoveryState::ToDelete, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x131) [0xaaf151]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0xa82b0b]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0xa72052]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x9ac6a4]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 11: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&)+0xd2) [0x9ac982]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x9a07dd]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xfc37bc]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xfc6970]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 15: (()+0x76ba) [0x7fe2557446ba]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 16: (clone()+0x6d) [0x7fe254d4b41d]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: <b>* Caught signal (Aborted) *<strong><br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: in thread 7fe236221700 thread_name:tp_osd_tp<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 1: (()+0x11390) [0x7fe25574e390]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2: (gsignal()+0x38) [0x7fe254c79428]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 3: (abort()+0x16a) [0x7fe254c7b02a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 4: (ceph::__ceph_abort(char const</strong>, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1a0) [0x850ac7]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x296a) [0xe44a2a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 6: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x5e6) [0xe48f96]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 7: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7f) [0xa0099f]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 8: (PG::_delete_some(ObjectStore::Transaction*)+0x710) [0xa64b30]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 9: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x71) [0xa658f1]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 10: (boost::statechart::simple_state<PG::RecoveryState::Deleting, PG::RecoveryState::ToDelete, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x131) [0xaaf151]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 11: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0xa82b0b]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 12: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0xa72052]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x9ac6a4]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 14: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&)+0xd2) [0x9ac982]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x9a07dd]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xfc37bc]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xfc6970]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 18: (()+0x76ba) [0x7fe2557446ba]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 19: (clone()+0x6d) [0x7fe254d4b41d]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2019-07-08 16:00:54.986 7fe236221700 -1 <strong></b> Caught signal (Aborted) *</strong><br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: in thread 7fe236221700 thread_name:tp_osd_tp<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 1: (()+0x11390) [0x7fe25574e390]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 2: (gsignal()+0x38) [0x7fe254c79428]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 3: (abort()+0x16a) [0x7fe254c7b02a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1a0) [0x850ac7]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x296a) [0xe44a2a]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 6: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x5e6) [0xe48f96]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 7: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7f) [0xa0099f]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 8: (PG::_delete_some(ObjectStore::Transaction*)+0x710) [0xa64b30]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 9: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x71) [0xa658f1]<br />Jul 08 16:00:54 synergy0 ceph-osd<sup><a href="#fn44682">44682</a></sup>: 10: (boost::statechart::simple_state<PG::RecoveryState::Deleting, PG::RecoveryState::ToDelete, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&...</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140258
2019-07-09T16:26:30Z
Edward Kalk
<ul></ul><p>I am confused by the "Copied to RADOS - Backport <a class="issue tracker-9 status-3 priority-4 priority-default closed" title="Backport: nautilus: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, ... (Resolved)" href="https://tracker.ceph.com/issues/39693">#39693</a>: nautilus" status. "Pending Backport 07/03/2019" <br />Does this mean it has been fixed and there is a patch or? Is there a workaround or code change I can insert?</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140308
2019-07-10T14:06:28Z
Edward Kalk
<ul></ul><p>will this be included in : <a class="external" href="https://tracker.ceph.com/projects/ceph/roadmap#v14.2.2">https://tracker.ceph.com/projects/ceph/roadmap#v14.2.2</a> . ?</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140314
2019-07-10T14:49:13Z
Edward Kalk
<ul><li><strong>File</strong> <a href="/attachments/download/4264/ceph-osd.5.clippedtoupload.log">ceph-osd.5.clippedtoupload.log</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/4264/ceph-osd.5.clippedtoupload.log">View</a> added</li><li><strong>File</strong> <a href="/attachments/download/4265/ceph-osd.21.clippedtoupload.log">ceph-osd.21.clippedtoupload.log</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/4265/ceph-osd.21.clippedtoupload.log">View</a> added</li></ul><p>We hit this bug again : "2019-07-10 09:16:27.728 7f73b844c700 -1 bluestore(/var/lib/ceph/osd/ceph-5) _txc_add_transaction error (39) Directory not empty not handled on operation 21" <br />This time it triggered when I added a OSD to a node.<br />Logs attached.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140317
2019-07-10T15:01:53Z
Edward Kalk
<ul></ul><p>found a few things that seem like fixes for this on github... : <a class="external" href="https://github.com/ceph/ceph/pull/27929/commits">https://github.com/ceph/ceph/pull/27929/commits</a></p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140438
2019-07-11T14:28:02Z
Edward Kalk
<ul></ul><p>OSD 9, 15, 10, 13 crashed this AM.<br /> ```ceph.log:2019-07-11 09:15:15.501601 mon.synergy0 (mon.0) 4248 : cluster [INF] osd.10 [v2:216.106.44.201:6802/25418,v1:216.106.44.201:6803/25418] boot<br />ceph.log:2019-07-11 09:15:20.561861 mon.synergy0 (mon.0) 4257 : cluster [INF] osd.13 [v2:216.106.44.201:6818/25427,v1:216.106.44.201:6820/25427] boot<br />ceph.log:2019-07-11 09:15:23.620936 mon.synergy0 (mon.0) 4263 : cluster [INF] osd.15 [v2:216.106.44.200:6802/17179,v1:216.106.44.200:6803/17179] boot<br />ceph.log:2019-07-11 09:15:39.440562 mon.synergy0 (mon.0) 4277 : cluster [INF] osd.9 [v2:216.106.44.200:6814/17315,v1:216.106.44.200:6815/17315] boot```</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140441
2019-07-11T14:28:55Z
Edward Kalk
<ul></ul><p>Was Bug 38724:<br /> ```ceph-osd.9.log: -3> 2019-07-11 09:15:13.569 7fc7b8243700 -1 bluestore(/var/lib/ceph/osd/ceph-9) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)<br />ceph-osd.15.log:2019-07-11 09:15:08.593 7fbe46b8b700 -1 bluestore(/var/lib/ceph/osd/ceph-15) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)<br />ceph-osd.13.log:2019-07-11 09:15:08.627 7fc575850700 -1 bluestore(/var/lib/ceph/osd/ceph-13) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)<br />ceph-osd.10.log:2019-07-11 09:15:08.543 7f0ef2a03700 -1 bluestore(/var/lib/ceph/osd/ceph-10) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)```</p>
<p>We are 14.2.1 on Ubuntu 16.04</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140444
2019-07-11T14:33:22Z
Edward Kalk
<ul></ul><p>Sometimes when this happens, the OSDs repeatedly crash and Linux system prevents them from being started. it takes 10 minuets for linux to allow them to start again. (too many start attempts) <br />15:24:13 synergy0 systemd<sup><a href="#fn1">1</a></sup>: <a class="email" href="mailto:ceph-osd@12.service">ceph-osd@12.service</a>: Start request repeated too quickly.<br />Jul 08 15:24:13 synergy0 systemd<sup><a href="#fn1">1</a></sup>: Failed to start Ceph object storage daemon osd.12.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140448
2019-07-11T14:36:12Z
Edward Kalk
<ul></ul><p>^^this results in the Production VMs becoming unresponsive as their disks are unavailable when we have multiple OSDs down on multiple hosts. (we are doing 2 copy) I've seen it where 3 OSDs are down at the same time on different hosts due to this bug. That's when we are seemingly really un lucky with the BUG. (3 copy would not have saved us from that)</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=140471
2019-07-11T16:39:49Z
Edward Kalk
<ul><li><strong>File</strong> <a href="/attachments/download/4267/osd%2010%20will%20not%20start%2007112019.txt">osd 10 will not start 07112019.txt</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/4267/osd%2010%20will%20not%20start%2007112019.txt">View</a> added</li></ul><p>Rebooted node 4, on node 1 and 2, 2 OSDs each crashed and will not start.<br />The logs are similar, seems to be the BUG related to 38724<br />Tried to manually start the OSDs, failed. It’s been about 20 minuets with prod down.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=141051
2019-07-19T18:24:24Z
Nathan Fish
<ul></ul><p>Edward Kalk wrote:</p>
<blockquote>
<p>Sometimes when this happens, the OSDs repeatedly crash and Linux system prevents them from being started. it takes 10 minuets for linux to allow them to start again. (too many start attempts) <br />15:24:13 synergy0 systemd<sup><a href="#fn1">1</a></sup>: <a class="email" href="mailto:ceph-osd@12.service">ceph-osd@12.service</a>: Start request repeated too quickly.<br />Jul 08 15:24:13 synergy0 systemd<sup><a href="#fn1">1</a></sup>: Failed to start Ceph object storage daemon osd.12.</p>
</blockquote>
<p>General systemd tip: you can reset the failure limit with 'systemctl reset-failed'. Then you can 'systemctl restart ceph-osd@12'.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=141458
2019-07-23T19:49:11Z
Nathan Fish
<ul></ul><p>It seems this will be in 14.2.3. When the fix comes out, will my crashed OSDs work again, or should I just purge and recreate them now?</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=142324
2019-08-02T16:34:36Z
Nathan Fish
<ul></ul><p>Just lost another OSD on 14.2.2. The cluster is still mostly empty, but a large parallel cp is going on, so rebuild will take a while. This is somewhat worrying. Any update on this?</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=142325
2019-08-02T18:14:58Z
Nathan Fish
<ul></ul><p>That OSD failure seems to have caused a cascade. Several more OSDs have crashed. 12% of objects were degraded, and I had to create new 'ssd' class OSDs to get enough failure domains. I cancelled the cp to prioritize recovery. Is there any workaround to repair the OSDs and get them to restart properly? They just crash again every time I restart them.</p>
<p>Can this bug please be set to a higher priority? This has caused an outage for myself and Edward above, and threatens data loss. That warrants at least Major.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=142706
2019-08-09T12:52:04Z
Florian Haas
<ul><li><strong>Affected Versions</strong> <i>v12.2.12</i> added</li></ul><p>So we're seeing a similar error after upgrading a Jewel cluster to Luminous (12.2.12), and then adding BlueStore OSDs to it:</p>
<pre>
-3> 2019-08-09 11:17:04.772976 7fb41585d700 -1 bluestore(/var/lib/ceph/osd/ceph-78) _txc_add_transaction error (7) Argument list too long not handled on operation 12 (op 2, counting from 0)
-2> 2019-08-09 11:17:04.772993 7fb41585d700 -1 bluestore(/var/lib/ceph/osd/ceph-78) unexpected error code
-1> 2019-08-09 11:17:04.773013 7fb41585d700 0 bluestore(/var/lib/ceph/osd/ceph-78) transaction dump:
{
"ops": [
{
"op_num": 0,
"op_name": "remove",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#"
},
{
"op_num": 1,
"op_name": "touch",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#"
},
{
"op_num": 2,
"op_name": "truncate",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#",
"offset": 4457615932
},
{
"op_num": 3,
"op_name": "op_setallochint",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#",
"expected_object_size": "0",
"expected_write_size": "0"
},
{
"op_num": 4,
"op_name": "write",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#",
"length": 8388608,
"offset": 0,
"bufferlist length": 8388608
},
{
"op_num": 5,
"op_name": "setattrs",
"collection": "2.293_head",
"oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#",
"attr_lens": {
"_": 254,
"snapset": 31
}
}
]
}
</pre>
<p>In our case, that's an E2BIG as opposed to ENOTEMPTY, but it does look awfully similar, at least superficially.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=142712
2019-08-09T14:36:39Z
Sage Weil
sage@newdream.net
<ul></ul><p>Florian Haas wrote:</p>
<blockquote>
<p>So we're seeing a similar error after upgrading a Jewel cluster to Luminous (12.2.12), and then adding BlueStore OSDs to it:</p>
<p>[...]</p>
<p>In our case, that's an E2BIG as opposed to ENOTEMPTY, but it does look awfully similar, at least superficially.</p>
</blockquote>
<p>BlueStore returns E2BIG because the object is > 4GB, which is not supported (size is stored in an u32). This cluster has some big rados objects.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=146945
2019-09-24T21:50:50Z
Brad Hubbard
bhubbard@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p>Bumping priority based on community feedback.</p>
RADOS - Bug #38724: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
https://tracker.ceph.com/issues/38724?journal_id=183532
2021-01-27T19:12:51Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul><p>While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".</p>