Ceph » RADOS

Raising priority because this is readily reproducible in baseline jewel. (By "baseline jewel" I mean jewel HEAD, currently at

This might be of interest, from /a/teuthology-2017-12-09_02:00:03-rados-jewel-distro-basic-smithi/1946610

2017-12-13T17:49:31.214 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 372 v 329 (ObjNum 371 snap 0 seq_num 371) dirty exists
2017-12-13T17:49:31.214 INFO:tasks.rados.rados.0.smithi097.stderr:Error: oid 372 write returned error code -34
2017-12-13T17:49:31.214 INFO:tasks.rados.rados.0.smithi097.stderr:Error: oid 372 write returned error code -34
2017-12-13T17:49:31.214 INFO:tasks.rados.rados.0.smithi097.stderr:Error: racing read on 372 returned version 0 rather than version 329
2017-12-13T17:49:31.214 INFO:tasks.rados.rados.0.smithi097.stderr:./test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f4041ffb700 time 2017-12-13 17:49:31.378082
2017-12-13T17:49:31.215 INFO:tasks.rados.rados.0.smithi097.stderr:./test/osd/RadosModel.h: 892: FAILED assert(0 == "racing read got wrong version")
2017-12-13T17:49:31.215 INFO:tasks.rados.rados.0.smithi097.stderr: ceph version 10.2.10-50-g1557449 (1557449a53febd15f448ce5acd354a3f67e0c0ba)
2017-12-13T17:49:31.215 INFO:tasks.rados.rados.0.smithi097.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x5628f1ecff10]
2017-12-13T17:49:31.215 INFO:tasks.rados.rados.0.smithi097.stderr: 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x4c3) [0x5628f1eaee03]
2017-12-13T17:49:31.215 INFO:tasks.rados.rados.0.smithi097.stderr: 3: (write_callback(void*, void*)+0x19) [0x5628f1ec7779]
2017-12-13T17:49:31.216 INFO:tasks.rados.rados.0.smithi097.stderr: 4: (()+0x9711d) [0x7f404f0ad11d]
2017-12-13T17:49:31.216 INFO:tasks.rados.rados.0.smithi097.stderr: 5: (()+0x81d79) [0x7f404f097d79]
2017-12-13T17:49:31.216 INFO:tasks.rados.rados.0.smithi097.stderr: 6: (()+0x171ace) [0x7f404f187ace]
2017-12-13T17:49:31.216 INFO:tasks.rados.rados.0.smithi097.stderr: 7: (()+0x76ba) [0x7f404e9d66ba]

In other words, the "racing read got wrong version" assert is tripped instead of the "finished tid X when last_acked_tid_was Y" assert. (Both asserts are in the same if (waiting_on == 0) conditional block.)

Here's a list of the affected tests:

[1946733] rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} hobj-sort.yaml msgr-failures/few.yaml msgr/simple.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/pggrow.yaml workloads/snaps-few-objects.yaml}

[1946550] rados/thrash-erasure-code/{clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/morepggrow.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml}

[1946659] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} hobj-sort.yaml msgr-failures/fastclose.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/pggrow.yaml workloads/small-objects.yaml}

[1946560] rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml supported/centos_latest.yaml thrashers/pggrow.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}

[1946610] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} hobj-sort.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/default.yaml workloads/cache.yaml}

[1946540] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} hobj-sort.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/mapgap.yaml workloads/cache.yaml}

[1946505] rados/upgrade/{hammer-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-hammer-install/hammer.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 6-next-mon/monb.yaml 7-workload/{radosbench.yaml rbd_api.yaml} 8-next-mon/monc.yaml 9-workload/{ec-rados-plugin=jerasure-k=3-m=1.yaml rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml test_cache-pool-snaps.yaml}} rados.yaml}

[1946624] rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml supported/centos_latest.yaml thrashers/morepggrow.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}

[1946742] rados/thrash-erasure-code/{clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/fastread.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}

So Nathan seems to have narrowed it down to https://github.com/ceph/ceph/pull/17815 - can you look at this when you're back David?