Bug #11660
Updated by Loïc Dachary almost 9 years ago
h3. Workaround
Update the timezone of the machines involved in the tests to be the same. See http://tracker.ceph.com/issues/11777
h3. Description
The utime_t->hitset archive name is timezone dependent. It should not be and if two OSDs are running on machines that do not have the same timezone, this bug will show up.
Hammer v0.94.2 release
Run: http://pulpito.ceph.redhat.com/teuthology-2015-05-15_16:15:16-rados-hammer-distro-basic-magna/
Jobs: ['109089', '109090', '109096', '109134']
Logs: http://pulpito.ceph.redhat.com/teuthology-2015-05-15_16:15:16-rados-hammer-distro-basic-magna/109089/
<pre>
2015-05-15T17:26:58.490 INFO:tasks.rados.rados.0.magna015.stdout:update_object_version oid 5026 v 7748 (ObjNum 5025 snap 0 seq_num 5025) dirty exists
2015-05-15T17:26:58.491 INFO:tasks.rados.rados.0.magna015.stdout:5026: left oid 5026 (ObjNum 5025 snap 0 seq_num 5025)
2015-05-15T17:26:58.531 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 1 to magna0159085-5027
2015-05-15T17:26:58.532 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 2 to magna0159085-5027
2015-05-15T17:26:58.538 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 3 to magna0159085-5027
2015-05-15T17:26:58.539 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 4 to magna0159085-5027
2015-05-15T17:26:58.580 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 6 to magna0159085-5027
2015-05-15T17:26:58.580 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 7 to magna0159085-5027
2015-05-15T17:26:58.584 INFO:tasks.rados.rados.0.magna015.stdout:update_object_version oid 5027 v 7754 (ObjNum 5026 snap 0 seq_num 5026) dirty exists
2015-05-15T17:26:58.624 INFO:tasks.rados.rados.0.magna015.stdout:5027: left oid 5027 (ObjNum 5026 snap 0 seq_num 5026)
2015-05-15T17:26:58.625 INFO:tasks.rados.rados.0.magna015.stdout:5029: finishing write tid 1 to magna0159085-5029
2015-05-15T17:26:58.630 INFO:tasks.rados.rados.0.magna015.stdout:5029: finishing write tid 2 to magna0159085-5029
2015-05-15T17:26:58.631 INFO:teuthology.orchestra.run.magna015.stderr:dumped all in format json
2015-05-15T17:26:58.681 INFO:tasks.ceph.osd.5.magna118.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7ff40fb3e700 time 2015-05-15 14:26:53.879218
2015-05-15T17:26:58.716 INFO:tasks.ceph.osd.5.magna118.stderr:osd/ReplicatedPG.cc: 10405: FAILED assert(obc)
2015-05-15T17:26:58.717 INFO:tasks.ceph.osd.5.magna118.stderr: ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b)
2015-05-15T17:26:58.719 INFO:tasks.ceph.osd.5.magna118.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645]
2015-05-15T17:26:58.720 INFO:tasks.ceph.osd.5.magna118.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x75f) [0x87149f]
2015-05-15T17:26:58.761 INFO:tasks.ceph.osd.5.magna118.stderr: 3: (ReplicatedPG::hit_set_persist()+0xff4) [0x8725d4]
2015-05-15T17:26:58.761 INFO:tasks.ceph.osd.5.magna118.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe3a) [0x890a9a]
2015-05-15T17:26:58.764 INFO:tasks.ceph.osd.5.magna118.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x83011a]
2015-05-15T17:26:58.765 INFO:tasks.ceph.osd.5.magna118.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x694685]
2015-05-15T17:26:58.805 INFO:tasks.ceph.osd.5.magna118.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x694ba9]
2015-05-15T17:26:58.807 INFO:tasks.ceph.osd.5.magna118.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x86f) [0xbb529f]
2015-05-15T17:26:58.810 INFO:tasks.ceph.osd.5.magna118.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xbb73d0]
2015-05-15T17:26:58.811 INFO:tasks.ceph.osd.5.magna118.stderr: 10: (()+0x7df3) [0x7ff42962fdf3]
2015-05-15T17:26:58.851 INFO:tasks.ceph.osd.5.magna118.stderr: 11: (clone()+0x6d) [0x7ff4281123dd]
2015-05-15T17:26:58.852 INFO:tasks.ceph.osd.5.magna118.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>