Bug #42175
closed_txc_add_transaction error (2) No such file or directory not handled on operation 15
0%
Description
2019-10-02T22:47:02.700 INFO:tasks.rados.rados.0.smithi191.stdout:1328: snap_create
2019-10-02T22:47:02.708 INFO:tasks.ceph.osd.2.smithi191.stderr:2019-10-02T22:47:02.674+0000 7ffbdadf7700 -1 received signal: Hangup from /usr/bin/python /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 14830) UID: 0
2019-10-02T22:47:02.719 INFO:tasks.rados.rados.0.smithi191.stdout:1326: expect (ObjNum 475 snap 107 seq_num 475)
2019-10-02T22:47:02.735 INFO:tasks.ceph.osd.1.smithi191.stderr:2019-10-02T22:47:02.702+0000 7f85d5e98700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _txc_add_transaction error (2) No such file or directory not handled on operation 15 (op 0, counting from 0)
2019-10-02T22:47:02.735 INFO:tasks.ceph.osd.1.smithi191.stderr:2019-10-02T22:47:02.702+0000 7f85d5e98700 -1 bluestore(/var/lib/ceph/osd/ceph-1) unexpected error code
2019-10-02T22:47:02.739 INFO:tasks.ceph.osd.1.smithi191.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5729-g804458b/rpm/el7/BUILD/ceph-15.0.0-5729-g804458b/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f85d5e98700 time 2019-10-02T22:47:02.703425+0000
2019-10-02T22:47:02.739 INFO:tasks.ceph.osd.1.smithi191.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5729-g804458b/rpm/el7/BUILD/ceph-15.0.0-5729-g804458b/src/os/bluestore/BlueStore.cc: 12589: ceph_abort_msg("unexpected error")
2019-10-02T22:47:02.739 INFO:tasks.ceph.osd.1.smithi191.stderr: ceph version 15.0.0-5729-g804458b (804458bf51d672c58fe3cb0cb33d91c185ac73e9) octopus (dev)
2019-10-02T22:47:02.739 INFO:tasks.ceph.osd.1.smithi191.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55b8e8529d9a]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0xcb0) [0x55b8e8ab0cb0]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 3: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x55b8e8accfe0]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55b8e8794614]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 5: (ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xc38) [0x55b8e891cb48]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 6: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x257) [0x55b8e892c077]
2019-10-02T22:47:02.740 INFO:tasks.ceph.osd.1.smithi191.stderr: 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x55b8e87c92ba]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5cb) [0x55b8e876e63b]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x2f9) [0x55b8e860ae09]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x55b8e8851cd9]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x143a) [0x55b8e862741a]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55b8e8bfb566]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b8e8bfe0b0]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 14: (()+0x7ea5) [0x7f85fa919ea5]
2019-10-02T22:47:02.741 INFO:tasks.ceph.osd.1.smithi191.stderr: 15: (clone()+0x6d) [0x7f85f97dd8cd]
Both jobs went dead, so no logs.
rados:thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml}
/a/nojha-2019-10-02_21:21:43-rados:thrash-master-distro-basic-smithi/4352410/
rados:thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}
/a/nojha-2019-10-02_21:21:43-rados:thrash-master-distro-basic-smithi/4352415/
Note that it appears with both more-active-recovery.yaml and more-async-partial-recovery.yaml(newly introduced).
We need logs to verify if this is a dup of https://tracker.ceph.com/issues/40522 or https://tracker.ceph.com/issues/37969.