Ceph

2014-08-24T10:31:01.379 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active!
2014-08-24T10:31:01.380 INFO:teuthology.orchestra.run.burnupi33:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph tell osd.0 flush_pg_stats'
2014-08-24T10:31:01.486 INFO:teuthology.orchestra.run.burnupi33:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph tell osd.2 flush_pg_stats'
2014-08-24T10:31:01.594 INFO:teuthology.orchestra.run.burnupi33:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty'
2014-08-24T10:31:01.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.135.24:6789/0'}, {u'name': u'b', u'rank': 1, 
u'addr': u'10.214.135.24:6790/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.135.24:6791/0'}], u'modified': u'2014-08-24 10:28:28.102217', u'fsid': u'd780d717-3ced-48dc-be0e-0a0fe5bae1bd', u'created': u'2014-08-24 10:28:28.102217'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoc
h': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.001008', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.002810', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'hea
lth_services': [{u'mons': [{u'last_updated': u'2014-08-24 10:30:32.223012', u'name': u'a', u'avail_percent': 94, u'kb_total': 944268724, u'kb_avail': 888579120, u'health': u'HEALTH_OK', u'kb_used': 7700408, u'store_stats': {u'bytes_total': 428989, u'bytes_log': 424183, u'last_updated': u'0.000000', u'bytes_misc': 4
806, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-24 10:30:32.227053', u'name': u'b', u'avail_percent': 94, u'kb_total': 944268724, u'kb_avail': 888579120, u'health': u'HEALTH_OK', u'kb_used': 7700408, u'store_stats': {u'bytes_total': 624031, u'bytes_log': 619225, u'last_updated': u'0.000000', u'bytes_misc': 4806
, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-24 10:30:32.436684', u'name': u'c', u'avail_percent': 94, u'kb_total': 944268724, u'kb_avail': 888579016, u'health': u'HEALTH_OK', u'kb_used': 7700512, u'store_stats': {u'bytes_total': 652174, u'bytes_log': 647368, u'last_updated': u'0.000000', u'bytes_misc': 4806, u
'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'28 pgs degraded'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs s
tuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs undersized'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 55/56 objects degraded (98.214%); 27/28 unfound (96.429%)'}]}, u'pgmap': {u'bytes_total': 1999307276288, u'degraded_objects': 55, u'num_pgs': 28, u'data_bytes': 16940, u'degraded_tot
al': 56, u'bytes_used': 212553728, u'unfound_ratio': u'96.429', u'unfound_objects': 27, u'version': 31, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+undersized+degraded'}, {u'count': 4, u'state_name': u'active+recovery_wait+undersized+degraded'}, {u'count': 10, u'state_name': u'active+recovering+undersi
zed+degraded'}], u'degraded_ratio': u'98.214', u'bytes_avail': 1996923547648, u'unfound_total': 28}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 3, u'num_up_osds': 2, u'epoch': 18, u'num_in_osds': 2}}, u'fsid': u'd780d717-3ced-48dc-be0e-0a0fe5bae1bd'
}
2014-08-24T10:31:01.772 INFO:tasks.rep_lost_unfound_delete:there are 27 unfound objects
2014-08-24T10:31:01.773 INFO:teuthology.orchestra.run.burnupi33:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2014-08-24T10:31:01.943 INFO:teuthology.orchestra.run.burnupi33.stderr:dumped all in format json
2014-08-24T10:31:01.957 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 1.e state active+recovering+undersized+degraded
2014-08-24T10:31:01.958 INFO:teuthology.orchestra.run.burnupi33:Running: u"adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -- pg 1.e list_missing '{}'" 
2014-08-24T10:31:02.121 INFO:teuthology.orchestra.run.burnupi33.stderr:Error ENOENT: i don't have pgid 1.e