Actions
Bug #11660
closed"osd/ReplicatedPG.cc: 10405: FAILED assert(obc)" in rados-hammer-distro-basic-magna
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Workaround¶
Update the timezone of the machines involved in the tests to be the same. See http://tracker.ceph.com/issues/11777
Description¶
The utime_t->hitset archive name is timezone dependent. It should not be and if two OSDs are running on machines that do not have the same timezone, this bug will show up.
Hammer v0.94.2 release
Run: http://pulpito.ceph.redhat.com/teuthology-2015-05-15_16:15:16-rados-hammer-distro-basic-magna/
Jobs: ['109089', '109090', '109096', '109134']
2015-05-15T17:26:58.490 INFO:tasks.rados.rados.0.magna015.stdout:update_object_version oid 5026 v 7748 (ObjNum 5025 snap 0 seq_num 5025) dirty exists 2015-05-15T17:26:58.491 INFO:tasks.rados.rados.0.magna015.stdout:5026: left oid 5026 (ObjNum 5025 snap 0 seq_num 5025) 2015-05-15T17:26:58.531 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 1 to magna0159085-5027 2015-05-15T17:26:58.532 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 2 to magna0159085-5027 2015-05-15T17:26:58.538 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 3 to magna0159085-5027 2015-05-15T17:26:58.539 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 4 to magna0159085-5027 2015-05-15T17:26:58.580 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 6 to magna0159085-5027 2015-05-15T17:26:58.580 INFO:tasks.rados.rados.0.magna015.stdout:5027: finishing write tid 7 to magna0159085-5027 2015-05-15T17:26:58.584 INFO:tasks.rados.rados.0.magna015.stdout:update_object_version oid 5027 v 7754 (ObjNum 5026 snap 0 seq_num 5026) dirty exists 2015-05-15T17:26:58.624 INFO:tasks.rados.rados.0.magna015.stdout:5027: left oid 5027 (ObjNum 5026 snap 0 seq_num 5026) 2015-05-15T17:26:58.625 INFO:tasks.rados.rados.0.magna015.stdout:5029: finishing write tid 1 to magna0159085-5029 2015-05-15T17:26:58.630 INFO:tasks.rados.rados.0.magna015.stdout:5029: finishing write tid 2 to magna0159085-5029 2015-05-15T17:26:58.631 INFO:teuthology.orchestra.run.magna015.stderr:dumped all in format json 2015-05-15T17:26:58.681 INFO:tasks.ceph.osd.5.magna118.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7ff40fb3e700 time 2015-05-15 14:26:53.879218 2015-05-15T17:26:58.716 INFO:tasks.ceph.osd.5.magna118.stderr:osd/ReplicatedPG.cc: 10405: FAILED assert(obc) 2015-05-15T17:26:58.717 INFO:tasks.ceph.osd.5.magna118.stderr: ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b) 2015-05-15T17:26:58.719 INFO:tasks.ceph.osd.5.magna118.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645] 2015-05-15T17:26:58.720 INFO:tasks.ceph.osd.5.magna118.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x75f) [0x87149f] 2015-05-15T17:26:58.761 INFO:tasks.ceph.osd.5.magna118.stderr: 3: (ReplicatedPG::hit_set_persist()+0xff4) [0x8725d4] 2015-05-15T17:26:58.761 INFO:tasks.ceph.osd.5.magna118.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe3a) [0x890a9a] 2015-05-15T17:26:58.764 INFO:tasks.ceph.osd.5.magna118.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x68a) [0x83011a] 2015-05-15T17:26:58.765 INFO:tasks.ceph.osd.5.magna118.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x694685] 2015-05-15T17:26:58.805 INFO:tasks.ceph.osd.5.magna118.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x2e9) [0x694ba9] 2015-05-15T17:26:58.807 INFO:tasks.ceph.osd.5.magna118.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x86f) [0xbb529f] 2015-05-15T17:26:58.810 INFO:tasks.ceph.osd.5.magna118.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xbb73d0] 2015-05-15T17:26:58.811 INFO:tasks.ceph.osd.5.magna118.stderr: 10: (()+0x7df3) [0x7ff42962fdf3] 2015-05-15T17:26:58.851 INFO:tasks.ceph.osd.5.magna118.stderr: 11: (clone()+0x6d) [0x7ff4281123dd] 2015-05-15T17:26:58.852 INFO:tasks.ceph.osd.5.magna118.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Files
Updated by Loïc Dachary almost 9 years ago
Maybe related to http://tracker.ceph.com/issues/9732 ?
Updated by Irek Fasikhov almost 9 years ago
Perhaps these problems are similar: http://tracker.ceph.com/issues/11511
Updated by Yuri Weinstein almost 9 years ago
Run: http://pulpito.ceph.redhat.com/teuthology-2015-05-20_17:15:43-rados-hammer-distro-basic-magna/
Jobs: ['112528', '112554', '112591']
Assertion: osd/ReplicatedPG.cc: 10405: FAILED assert(obc) ceph version 0.94.1-116-g63832d4 (63832d4039889b6b704b88b86eaba4aadcfceb2e) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc2bcb] 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x72f) [0x881b8f] 3: (ReplicatedPG::hit_set_persist()+0xfcc) [0x882c7c] 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe2e) [0x8a3b2e] 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x84149a] 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x6a0f85] 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x331) [0x6a14d1] 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x875) [0xbb2cc5] 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xbb4de0] 10: (()+0x8182) [0x7fdc10085182] 11: (clone()+0x6d) [0x7fdc0e5effbd]
Updated by Loïc Dachary almost 9 years ago
- Status changed from New to In Progress
- Assignee set to Loïc Dachary
Updated by Loïc Dachary almost 9 years ago
- rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/cache-agent-small.yaml}
- rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/cache-agent-small.yaml}
- rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/fastclose.yaml thrashers/mapgap.yaml workloads/cache-agent-small.yaml}
Updated by Loïc Dachary almost 9 years ago
20 runs did not repeat the failure
filter='rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/cache-agent-small.yaml}'
./virtualenv/bin/teuthology-suite --num 10 --priority 101 --filter="$filter" --suite rados --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph hammer
Updated by Loïc Dachary almost 9 years ago
log of osd 5
-271> 2015-05-20 18:38:23.841762 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:33:10.789855_2015-05-20 18:34:12.870678/head/.ceph-internal/2
-270> 2015-05-20 18:38:23.841781 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-269> 2015-05-20 18:38:23.841823 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] hit_set_persist archive 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-268> 2015-05-20 18:38:23.841842 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-267> 2015-05-20 18:38:23.841859 7fdbf47de700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 '_'
-266> 2015-05-20 18:38:23.841927 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) error opening file /var/lib/ceph/osd/ceph-5/current/2.1_head/hit\uset\u2.1\uarchive\u2015-05-20 18:37:22.769797\u2015-05-20 18:38:23.841578__head_00000001_.ceph-internal_2 with flags=2: (2) No such file or directory
-265> 2015-05-20 18:38:23.841946 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 '_' = -2
-264> 2015-05-20 18:38:23.841951 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 but can_create
-263> 2015-05-20 18:38:23.841968 7fdbf47de700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 'snapset'
-262> 2015-05-20 18:38:23.841996 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) error opening file /var/lib/ceph/osd/ceph-5/current/2.1_head/hit\uset\u2.1\uarchive\u2015-05-20 18:37:22.769797\u2015-05-20 18:38:23.841578__head_00000001_.ceph-internal_2 with flags=2: (2) No such file or directory
-261> 2015-05-20 18:38:23.842006 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 'snapset' = -2
-260> 2015-05-20 18:38:23.842011 7fdbf47de700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/snapdir/.ceph-internal/2 'snapset'
-259> 2015-05-20 18:38:23.842038 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) error opening file /var/lib/ceph/osd/ceph-5/current/2.1_head/hit\uset\u2.1\uarchive\u2015-05-20 18:37:22.769797\u2015-05-20 18:38:23.841578__snapdir_00000001_.ceph-internal_2 with flags=2: (2) No such file or directory
-258> 2015-05-20 18:38:23.842045 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/snapdir/.ceph-internal/2 'snapset' = -2
-256> 2015-05-20 18:38:23.842093 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] create_object_context 0x67ca3c0 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-253> 2015-05-20 18:38:23.842129 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] populate_obc_watchers 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-251> 2015-05-20 18:38:23.842141 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-245> 2015-05-20 18:38:23.842150 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_object_context: 0x67ca3c0 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2 rwstate(none n=0 w=0) oi: 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv 0) ssc: 0x6086ff0 snapset: 0=[]:[]
-243> 2015-05-20 18:38:23.842164 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] simple_repop_create 1/hit_set_2.1_archive_2015-05-20 18:37:22.769797_2015-05-20 18:38:23.841578/head/.ceph-internal/2
-240> 2015-05-20 18:38:23.842178 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] new_repop rep_tid 1006 (no op)
-228> 2015-05-20 18:38:23.842213 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] hit_set_create bloom{false_positive_probability: 0.05, target_size: 0, seed: 0}
-223> 2015-05-20 18:38:23.842238 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] hit_set_create previous set had approx 120 unique items over 61.072410 seconds
-219> 2015-05-20 18:38:23.842252 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] hit_set_create target_size 1000 fpp 0.00625
-197> 2015-05-20 18:38:23.842496 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2
-194> 2015-05-20 18:38:23.842507 7fdbf47de700 20 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] hit_set_trim removing 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2
-193> 2015-05-20 18:38:23.842521 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2
-192> 2015-05-20 18:38:23.842530 7fdbf47de700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_'
-189> 2015-05-20 18:38:23.842556 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) error opening file /var/lib/ceph/osd/ceph-5/current/2.1_head/hit\uset\u2.1\uarchive\u2015-05-20 18:21:56.078514\u2015-05-20 18:22:58.416980__head_00000001_.ceph-internal_2 with flags=2: (2) No such file or directory
-188> 2015-05-20 18:38:23.842563 7fdbf47de700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2
-187> 2015-05-20 18:38:23.842565 7fdbf47de700 10 osd.5 pg_epoch: 352 pg[2.1( v 351'4090 (20'1001,351'4090] local-les=333 n=58 ec=6 les/c 333/338 332/332/316) [5,2] r=0 lpr=332 crt=350'4088 lcod 350'4089 mlcod 350'4089 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create
0> 2015-05-20 18:38:23.860575 7fdbf47de700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7fdbf47de700 time 2015-05-20 18:38:23.842575
osd/ReplicatedPG.cc: 10405: FAILED assert(obc)
Updated by Loïc Dachary almost 9 years ago
The log lines containing the OID 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 which is the one that was not found in the cache.
2015-05-20 18:36:07.117205 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3808 (14'664,319'3808] local-les=317 n=61 ec=6 les/c 317/315 316/316/316) [5,2] r=0 lpr=316 pi=108-315/23 luod=317'3803 lua=317'3803 rops=3 crt=317'3802 lcod 317'3803 mlcod 317'3802 active+recovering+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:07.117217 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3808 (14'664,319'3808] local-les=317 n=61 ec=6 les/c 317/315 316/316/316) [5,2] r=0 lpr=316 pi=108-315/23 luod=317'3803 lua=317'3803 rops=3 crt=317'3802 lcod 317'3803 mlcod 317'3802 active+recovering+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:07.117226 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:07.117273 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:07.117276 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3808 (14'664,319'3808] local-les=317 n=61 ec=6 les/c 317/315 316/316/316) [5,2] r=0 lpr=316 pi=108-315/23 luod=317'3803 lua=317'3803 rops=3 crt=317'3802 lcod 317'3803 mlcod 317'3802 active+recovering+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:07.117287 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3808 (14'664,319'3808] local-les=317 n=61 ec=6 les/c 317/315 316/316/316) [5,2] r=0 lpr=316 pi=108-315/23 luod=317'3803 lua=317'3803 rops=3 crt=317'3802 lcod 317'3803 mlcod 317'3802 active+recovering+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.202444 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3808 crt=317'3802 lcod 319'3812 mlcod 319'3811 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.202455 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3808 crt=317'3802 lcod 319'3812 mlcod 319'3811 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.202465 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:09.202511 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:09.202514 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3808 crt=317'3802 lcod 319'3812 mlcod 319'3811 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:09.202524 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3808 crt=317'3802 lcod 319'3812 mlcod 319'3811 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.203583 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3809 crt=317'3802 lcod 319'3812 mlcod 319'3812 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.203593 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3809 crt=317'3802 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.203680 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:09.203712 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:09.203715 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3809 crt=317'3802 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:09.203724 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3813 (14'664,319'3813] local-les=317 n=63 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3809 crt=317'3802 lcod 319'3812 mlcod 319'3812 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.699762 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.699776 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.699787 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:09.699833 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:09.699836 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:09.699846 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.701095 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.701107 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:09.701116 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:09.701161 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:09.701164 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:09.701174 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3814 (14'664,319'3814] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3813 lua=319'3810 crt=319'3812 lcod 319'3812 mlcod 319'3812 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.090105 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3818 (14'664,319'3818] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=319'3813 lcod 319'3817 mlcod 319'3817 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.090116 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3818 (14'664,319'3818] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=319'3813 lcod 319'3817 mlcod 319'3817 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.090125 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:10.090175 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:10.090178 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3818 (14'664,319'3818] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=319'3813 lcod 319'3817 mlcod 319'3817 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:10.090188 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3818 (14'664,319'3818] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=319'3813 lcod 319'3817 mlcod 319'3817 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.358498 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3819 (14'664,319'3819] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3818 lua=319'3818 crt=319'3817 lcod 319'3817 mlcod 319'3817 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.358513 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3819 (14'664,319'3819] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3818 lua=319'3818 crt=319'3817 lcod 319'3817 mlcod 319'3817 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.358525 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:10.358571 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:10.358575 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3819 (14'664,319'3819] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3818 lua=319'3818 crt=319'3817 lcod 319'3817 mlcod 319'3817 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:10.358585 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3819 (14'664,319'3819] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3818 lua=319'3818 crt=319'3817 lcod 319'3817 mlcod 319'3817 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.460062 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3821 (14'664,319'3821] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3820 lua=319'3818 crt=319'3817 lcod 319'3819 mlcod 319'3817 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.460073 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3821 (14'664,319'3821] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3820 lua=319'3818 crt=319'3817 lcod 319'3819 mlcod 319'3817 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:10.460082 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:10.460122 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:10.460125 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3821 (14'664,319'3821] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3820 lua=319'3818 crt=319'3817 lcod 319'3819 mlcod 319'3817 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:10.460134 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3821 (14'664,319'3821] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3820 lua=319'3818 crt=319'3817 lcod 319'3819 mlcod 319'3817 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.366228 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3822 (14'664,319'3822] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3819 crt=319'3817 lcod 319'3821 mlcod 319'3820 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.366239 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3822 (14'664,319'3822] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3819 crt=319'3817 lcod 319'3821 mlcod 319'3820 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.366249 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:11.366296 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:11.366299 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3822 (14'664,319'3822] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3819 crt=319'3817 lcod 319'3821 mlcod 319'3820 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:11.366312 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3822 (14'664,319'3822] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=319'3819 crt=319'3817 lcod 319'3821 mlcod 319'3820 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.370500 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3823 (14'664,319'3823] local-les=317 n=58 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3822 lua=319'3820 crt=319'3820 lcod 319'3821 mlcod 319'3821 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.370512 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3823 (14'664,319'3823] local-les=317 n=58 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3822 lua=319'3820 crt=319'3820 lcod 319'3821 mlcod 319'3821 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:11.370523 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:11.370562 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:11.370565 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3823 (14'664,319'3823] local-les=317 n=58 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3822 lua=319'3820 crt=319'3820 lcod 319'3821 mlcod 319'3821 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:11.370576 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3823 (14'664,319'3823] local-les=317 n=58 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3822 lua=319'3820 crt=319'3820 lcod 319'3821 mlcod 319'3821 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:17.909524 7fdbebecc700 20 osd.5 pg_epoch: 319 pg[2.1( v 319'3834 (18'826,319'3834] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3832 lua=319'3832 crt=319'3831 lcod 319'3831 mlcod 319'3831 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:17.909546 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3834 (18'826,319'3834] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3832 lua=319'3832 crt=319'3831 lcod 319'3831 mlcod 319'3831 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:17.909563 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:17.909650 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:17.909655 7fdbebecc700 10 osd.5 pg_epoch: 319 pg[2.1( v 319'3834 (18'826,319'3834] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3832 lua=319'3832 crt=319'3831 lcod 319'3831 mlcod 319'3831 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:17.909672 7fdbebecc700 -1 osd.5 pg_epoch: 319 pg[2.1( v 319'3834 (18'826,319'3834] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=319'3832 lua=319'3832 crt=319'3831 lcod 319'3831 mlcod 319'3831 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.533324 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 320'3858 (18'826,320'3858] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3850 crt=319'3836 lcod 320'3857 mlcod 320'3857 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.533337 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 320'3858 (18'826,320'3858] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3850 crt=319'3836 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.533347 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.534477 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.534495 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 320'3858 (18'826,320'3858] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3850 crt=319'3836 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.534506 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 320'3858 (18'826,320'3858] local-les=317 n=62 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3850 crt=319'3836 lcod 320'3857 mlcod 320'3857 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.540380 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3859 (18'826,321'3859] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3850 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.540402 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3859 (18'826,321'3859] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3850 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.540423 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.540487 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.540492 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3859 (18'826,321'3859] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3850 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.540513 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3859 (18'826,321'3859] local-les=317 n=61 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3850 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.548791 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3853 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.548801 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3853 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.548809 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.548881 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.548884 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3853 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.548894 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=320'3858 lua=320'3853 crt=320'3857 lcod 320'3857 mlcod 320'3857 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.628571 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.628586 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.628596 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.628649 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.628652 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.628663 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.629627 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.629649 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.629666 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.629714 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.629718 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.629738 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3860 (18'826,321'3860] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 lua=320'3858 crt=320'3857 lcod 321'3859 mlcod 321'3859 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.840754 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3861 (18'826,321'3861] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3860 lua=321'3860 crt=321'3859 lcod 321'3859 mlcod 321'3859 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.840767 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3861 (18'826,321'3861] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3860 lua=321'3860 crt=321'3859 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:21.840779 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:21.840820 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:21.840823 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3861 (18'826,321'3861] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3860 lua=321'3860 crt=321'3859 lcod 321'3859 mlcod 321'3859 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:21.840833 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3861 (18'826,321'3861] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3860 lua=321'3860 crt=321'3859 lcod 321'3859 mlcod 321'3859 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.356268 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.356288 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.356303 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:22.356366 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:22.356371 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:22.356385 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.357777 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.357791 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.357804 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:22.357860 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:22.357864 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:22.357879 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3864 lua=321'3863 crt=321'3860 lcod 321'3863 mlcod 321'3863 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.552360 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=321'3860 lcod 321'3865 mlcod 321'3865 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.552407 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=321'3860 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.552416 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:22.552462 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:22.552465 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=321'3860 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:22.552475 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3866 (18'826,321'3866] local-les=317 n=60 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 crt=321'3860 lcod 321'3865 mlcod 321'3865 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.558657 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3868 (18'826,321'3868] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.558669 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3868 (18'826,321'3868] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.558679 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:22.558732 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:22.558735 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3868 (18'826,321'3868] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:22.558745 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3868 (18'826,321'3868] local-les=317 n=59 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.564169 7fdbebecc700 20 osd.5 pg_epoch: 321 pg[2.1( v 321'3870 (18'826,321'3870] local-les=317 n=57 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.564178 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3870 (18'826,321'3870] local-les=317 n=57 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:22.564187 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:22.564219 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:22.564221 7fdbebecc700 10 osd.5 pg_epoch: 321 pg[2.1( v 321'3870 (18'826,321'3870] local-les=317 n=57 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:22.564231 7fdbebecc700 -1 osd.5 pg_epoch: 321 pg[2.1( v 321'3870 (18'826,321'3870] local-les=317 n=57 ec=6 les/c 317/319 316/316/316) [5,2] r=0 lpr=316 luod=321'3866 lua=321'3866 crt=321'3865 lcod 321'3865 mlcod 321'3865 active+clean] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:46.176342 7fdbebecc700 20 osd.5 pg_epoch: 324 pg[2.1( v 324'3886 (18'826,324'3886] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3885 lua=324'3885 crt=323'3884 lcod 323'3884 mlcod 323'3884 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:46.176352 7fdbebecc700 10 osd.5 pg_epoch: 324 pg[2.1( v 324'3886 (18'826,324'3886] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3885 lua=324'3885 crt=323'3884 lcod 323'3884 mlcod 323'3884 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:46.176361 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:46.176400 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:46.176403 7fdbebecc700 10 osd.5 pg_epoch: 324 pg[2.1( v 324'3886 (18'826,324'3886] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3885 lua=324'3885 crt=323'3884 lcod 323'3884 mlcod 323'3884 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:46.176413 7fdbebecc700 -1 osd.5 pg_epoch: 324 pg[2.1( v 324'3886 (18'826,324'3886] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3885 lua=324'3885 crt=323'3884 lcod 323'3884 mlcod 323'3884 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:48.711157 7fdbebecc700 20 osd.5 pg_epoch: 325 pg[2.1( v 325'3900 (18'826,325'3900] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3892 lua=324'3892 crt=324'3891 lcod 324'3891 mlcod 324'3891 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:48.711167 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3900 (18'826,325'3900] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3892 lua=324'3892 crt=324'3891 lcod 324'3891 mlcod 324'3891 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:48.711176 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:48.711221 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:48.711224 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3900 (18'826,325'3900] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3892 lua=324'3892 crt=324'3891 lcod 324'3891 mlcod 324'3891 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:48.711234 7fdbebecc700 -1 osd.5 pg_epoch: 325 pg[2.1( v 325'3900 (18'826,325'3900] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=324'3892 lua=324'3892 crt=324'3891 lcod 324'3891 mlcod 324'3891 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.804442 7fdbebecc700 20 osd.5 pg_epoch: 325 pg[2.1( v 325'3916 (18'826,325'3916] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 crt=325'3914 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.804457 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3916 (18'826,325'3916] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 crt=325'3914 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.804472 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:51.804537 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:51.804540 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3916 (18'826,325'3916] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 crt=325'3914 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:51.804550 7fdbebecc700 -1 osd.5 pg_epoch: 325 pg[2.1( v 325'3916 (18'826,325'3916] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 crt=325'3914 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.808997 7fdbebecc700 20 osd.5 pg_epoch: 325 pg[2.1( v 325'3917 (18'826,325'3917] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=325'3916 lua=325'3916 crt=325'3915 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.809006 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3917 (18'826,325'3917] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=325'3916 lua=325'3916 crt=325'3915 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:36:51.809015 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:36:51.809043 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:36:51.809045 7fdbebecc700 10 osd.5 pg_epoch: 325 pg[2.1( v 325'3917 (18'826,325'3917] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=325'3916 lua=325'3916 crt=325'3915 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:36:51.809054 7fdbebecc700 -1 osd.5 pg_epoch: 325 pg[2.1( v 325'3917 (18'826,325'3917] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=325'3916 lua=325'3916 crt=325'3915 lcod 325'3915 mlcod 325'3915 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:02.453525 7fdbebecc700 20 osd.5 pg_epoch: 328 pg[2.1( v 328'3933 (18'826,328'3933] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=328'3931 lua=328'3931 crt=328'3930 lcod 328'3930 mlcod 328'3930 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:02.453547 7fdbebecc700 10 osd.5 pg_epoch: 328 pg[2.1( v 328'3933 (18'826,328'3933] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=328'3931 lua=328'3931 crt=328'3930 lcod 328'3930 mlcod 328'3930 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:02.453564 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:02.453658 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:02.453664 7fdbebecc700 10 osd.5 pg_epoch: 328 pg[2.1( v 328'3933 (18'826,328'3933] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=328'3931 lua=328'3931 crt=328'3930 lcod 328'3930 mlcod 328'3930 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:02.453682 7fdbebecc700 -1 osd.5 pg_epoch: 328 pg[2.1( v 328'3933 (18'826,328'3933] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=328'3931 lua=328'3931 crt=328'3930 lcod 328'3930 mlcod 328'3930 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.649259 7fdbebecc700 20 osd.5 pg_epoch: 331 pg[2.1( v 331'3956 (18'826,331'3956] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 lua=331'3954 crt=331'3950 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.649271 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3956 (18'826,331'3956] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 lua=331'3954 crt=331'3950 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.649282 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:17.649352 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:17.649356 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3956 (18'826,331'3956] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 lua=331'3954 crt=331'3950 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:17.649367 7fdbebecc700 -1 osd.5 pg_epoch: 331 pg[2.1( v 331'3956 (18'826,331'3956] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 lua=331'3954 crt=331'3950 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.654377 7fdbebecc700 20 osd.5 pg_epoch: 331 pg[2.1( v 331'3957 (18'826,331'3957] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.654388 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3957 (18'826,331'3957] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.654397 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:17.654434 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:17.654437 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3957 (18'826,331'3957] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:17.654447 7fdbebecc700 -1 osd.5 pg_epoch: 331 pg[2.1( v 331'3957 (18'826,331'3957] local-les=323 n=58 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.658749 7fdbebecc700 20 osd.5 pg_epoch: 331 pg[2.1( v 331'3958 (18'826,331'3958] local-les=323 n=57 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.658759 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3958 (18'826,331'3958] local-les=323 n=57 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:17.658769 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:17.658805 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:17.658808 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3958 (18'826,331'3958] local-les=323 n=57 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:17.658818 7fdbebecc700 -1 osd.5 pg_epoch: 331 pg[2.1( v 331'3958 (18'826,331'3958] local-les=323 n=57 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3956 lua=331'3954 crt=331'3955 lcod 331'3955 mlcod 331'3955 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:19.534020 7fdbebecc700 20 osd.5 pg_epoch: 331 pg[2.1( v 331'3966 (18'826,331'3966] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3965 lua=331'3965 crt=331'3964 lcod 331'3964 mlcod 331'3964 active+undersized+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:19.534033 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3966 (18'826,331'3966] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3965 lua=331'3965 crt=331'3964 lcod 331'3964 mlcod 331'3964 active+undersized+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:19.534043 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:19.534096 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:19.534099 7fdbebecc700 10 osd.5 pg_epoch: 331 pg[2.1( v 331'3966 (18'826,331'3966] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3965 lua=331'3965 crt=331'3964 lcod 331'3964 mlcod 331'3964 active+undersized+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:19.534108 7fdbebecc700 -1 osd.5 pg_epoch: 331 pg[2.1( v 331'3966 (18'826,331'3966] local-les=323 n=59 ec=6 les/c 323/323 322/322/316) [5] r=0 lpr=322 luod=331'3965 lua=331'3965 crt=331'3964 lcod 331'3964 mlcod 331'3964 active+undersized+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.623249 7fdbebecc700 20 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.623262 7fdbebecc700 10 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.623275 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:26.623324 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:26.623329 7fdbebecc700 10 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:26.623343 7fdbebecc700 -1 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.625176 7fdbebecc700 20 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_hit_set_archive_object 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.625186 7fdbebecc700 10 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_object_context: obc NOT found in cache: 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:26.625198 7fdbebecc700 15 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' 2015-05-20 18:37:26.625234 7fdbebecc700 10 filestore(/var/lib/ceph/osd/ceph-5) getattr 2.1_head/1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 '_' = -2 2015-05-20 18:37:26.625237 7fdbebecc700 10 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] get_object_context: no obc for soid 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 and !can_create 2015-05-20 18:37:26.625247 7fdbebecc700 -1 osd.5 pg_epoch: 335 pg[2.1( v 335'3983 (18'826,335'3983] local-les=333 n=59 ec=6 les/c 333/323 332/332/316) [5,2] r=0 lpr=332 pi=322-331/1 luod=333'3981 lua=333'3981 rops=15 crt=331'3964 lcod 333'3980 mlcod 321'3881 active+recovering+degraded] agent_load_hit_sets: could not load hitset 1/hit_set_2.1_archive_2015-05-20 18:21:56.078514_2015-05-20 18:22:58.416980/head/.ceph-internal/2 2015-05-20 18:37:40.467945 7fdbebecc700 20 osd.5 pg_epoch: 340 pg[2.1( v 340'3997 (18'826,340'399
Updated by Samuel Just almost 9 years ago
(03:36:43 PM) sjustwork: these two nodes are 3 hours off.
(03:37:03 PM) sjustwork: the utime_t->string calculation is timezone dependent
(03:37:21 PM) sjustwork: loicd: confirm that in the other cases, some of the osds were on one timezone and the others were on the other
Updated by Samuel Just almost 9 years ago
- Status changed from In Progress to Duplicate
Updated by Yuri Weinstein almost 9 years ago
Run: http://pulpito.ceph.redhat.com/teuthology-2015-05-18_21:00:02-rados-hammer-distro-basic-magna/
Jobs: ['111030', '111055', '111076', '111092', '111108', '111160']
Actions
#15
Updated by CephM M almost 9 years ago
I start my osds. They crash. The osds that fail are part of an SSD cache.
I've attached a sample crash and my osd map.
Updated by Bernd Helm almost 9 years ago
i also hit that bug with hammer 0.94.2 on two osd's which are part of my SSD cache after updating the timezone on all machines.
so changing timezone on all osds and only restart some OSD seems to trigger this bug. it only happend on OSD's that actually contained data.
a full restart of ceph solved this for me.
Updated by Bernd Helm almost 9 years ago
Bernd Helm wrote:
a full restart of ceph solved this for me.
i need to correct me, osd's keep crashing. all have the same timezone and have been rebootet. so the workaround above is not working.
Updated by Bernd Helm almost 9 years ago
i have moved my servers back to the previous timezone because i could not get the cluster stable with the new one.
Actions