2014-08-08T10:38:54.998 INFO:teuthology.orchestra.run.vpm167.stderr:testUtf8Container (test.functional.tests.TestContainer) ... ok 2014-08-08T10:38:56.042 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T10:38:56.117 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T10:38:57.226 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T10:38:57.291 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T10:38:57.291 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:38:57.719 INFO:tasks.rados.rados.0.vpm167.stdout:2939: done (4 left) 2014-08-08T10:38:57.719 INFO:tasks.rados.rados.0.vpm167.stdout:2941: write oid 29 current snap is 391 2014-08-08T10:38:57.720 INFO:tasks.rados.rados.0.vpm167.stdout:2941: seq_num 1108 ranges {607032=409163,1648806=558226} 2014-08-08T10:38:57.758 INFO:tasks.rados.rados.0.vpm167.stdout:2941: writing vpm1675861-29 from 607032 to 1016195 tid 1 2014-08-08T10:38:57.805 INFO:tasks.rados.rados.0.vpm167.stdout:2941: writing vpm1675861-29 from 1648806 to 2207032 tid 2 2014-08-08T10:38:57.809 INFO:tasks.rados.rados.0.vpm167.stdout:2940: finishing write tid 1 to vpm1675861-34 2014-08-08T10:38:57.809 INFO:tasks.rados.rados.0.vpm167.stdout:2940: finishing write tid 2 to vpm1675861-34 2014-08-08T10:38:57.809 INFO:tasks.rados.rados.0.vpm167.stdout:2940: finishing write tid 4 to vpm1675861-34 2014-08-08T10:38:57.809 INFO:tasks.rados.rados.0.vpm167.stdout:2940: finishing write tid 5 to vpm1675861-34 2014-08-08T10:38:57.809 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1611 (ObjNum 1107 snap 391 seq_num 1107) dirty exists 2014-08-08T10:38:57.810 INFO:tasks.rados.rados.0.vpm167.stdout:2938: finishing write tid 1 to vpm1675861-16 2014-08-08T10:38:57.810 INFO:tasks.rados.rados.0.vpm167.stdout:2935: finishing write tid 1 to vpm1675861-20 2014-08-08T10:38:57.811 INFO:tasks.rados.rados.0.vpm167.stdout:2935: finishing write tid 3 to vpm1675861-20 2014-08-08T10:38:57.811 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 5 v 1102 (ObjNum 1079 snap 374 seq_num 1079) dirty exists 2014-08-08T10:38:57.811 INFO:tasks.rados.rados.0.vpm167.stdout:2935: finishing write tid 4 to vpm1675861-20 2014-08-08T10:38:57.811 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 20 v 869 (ObjNum 1105 snap 391 seq_num 1105) dirty exists 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2938: finishing write tid 2 to vpm1675861-16 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2938: finishing write tid 4 to vpm1675861-16 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2938: finishing write tid 5 to vpm1675861-16 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1346 (ObjNum 1106 snap 391 seq_num 1106) dirty exists 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2935: done (4 left) 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2937: done (3 left) 2014-08-08T10:38:57.812 INFO:tasks.rados.rados.0.vpm167.stdout:2938: done (2 left) 2014-08-08T10:38:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:2940: done (1 left) 2014-08-08T10:38:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:2942: read oid 41 snap -1 2014-08-08T10:38:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:2943: write oid 6 current snap is 391 2014-08-08T10:38:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:2943: seq_num 1109 ranges {418614=532392,1478267=506348,2418613=1} 2014-08-08T10:38:57.845 INFO:tasks.rados.rados.0.vpm167.stdout:2943: writing vpm1675861-6 from 418614 to 951006 tid 1 2014-08-08T10:38:57.888 INFO:tasks.rados.rados.0.vpm167.stdout:2943: writing vpm1675861-6 from 1478267 to 1984615 tid 2 2014-08-08T10:38:57.896 INFO:tasks.rados.rados.0.vpm167.stdout:2943: writing vpm1675861-6 from 2418613 to 2418614 tid 3 2014-08-08T10:38:57.897 INFO:tasks.rados.rados.0.vpm167.stdout:2941: finishing write tid 1 to vpm1675861-29 2014-08-08T10:38:57.897 INFO:tasks.rados.rados.0.vpm167.stdout:2944: delete oid 35 current snap is 391 2014-08-08T10:38:57.911 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:38:57.985 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:38:57.990 INFO:tasks.rados.rados.0.vpm167.stdout:2942: expect (ObjNum 1100 snap 385 seq_num 1100) 2014-08-08T10:38:58.351 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:38:58.419 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:38:58.963 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005534', u'skew': u'-0.000012', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.009411', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:38:19.604490', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 88843436, u'health': u'HEALTH_OK', u'kb_used': 10110132, u'store_stats': {u'bytes_total': 24928852, u'bytes_log': 3324925, u'last_updated': u'0.000000', u'bytes_misc': 21603927, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:38:26.391858', u'name': u'c', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 87730188, u'health': u'HEALTH_OK', u'kb_used': 11223380, u'store_stats': {u'bytes_total': 27345388, u'bytes_log': 5770279, u'last_updated': u'0.000000', u'bytes_misc': 21575109, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:38:23.348881', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 88838172, u'health': u'HEALTH_OK', u'kb_used': 10115396, u'store_stats': {u'bytes_total': 26777103, u'bytes_log': 5181873, u'last_updated': u'0.000000', u'bytes_misc': 21595230, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 308, u'data_bytes': 152435254, u'bytes_used': 1369899008, u'version': 2193, u'pgs_by_state': [{u'count': 308, u'state_name': u'active+clean'}], u'bytes_avail': 1286491144192}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1157, u'num_in_osds': 6}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:38:58.964 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T10:38:58.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:38:58.964 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 0, 5, 2, 4, 3] 2014-08-08T10:38:59.431 INFO:tasks.rados.rados.0.vpm167.stdout:2942: done (3 left) 2014-08-08T10:38:59.431 INFO:tasks.rados.rados.0.vpm167.stdout:2944: done (2 left) 2014-08-08T10:38:59.432 INFO:tasks.rados.rados.0.vpm167.stdout:2945: write oid 35 current snap is 391 2014-08-08T10:38:59.432 INFO:tasks.rados.rados.0.vpm167.stdout:2945: seq_num 1111 ranges {707250=512096,1955107=646738,3152577=680459,3907249=1} 2014-08-08T10:38:59.464 INFO:tasks.rados.rados.0.vpm167.stdout:2945: writing vpm1675861-35 from 707250 to 1219346 tid 1 2014-08-08T10:38:59.508 INFO:tasks.rados.rados.0.vpm167.stdout:2945: writing vpm1675861-35 from 1955107 to 2601845 tid 2 2014-08-08T10:38:59.550 INFO:tasks.rados.rados.0.vpm167.stdout:2945: writing vpm1675861-35 from 3152577 to 3833036 tid 3 2014-08-08T10:38:59.553 INFO:tasks.rados.rados.0.vpm167.stdout:2945: writing vpm1675861-35 from 3907249 to 3907250 tid 4 2014-08-08T10:38:59.554 INFO:tasks.rados.rados.0.vpm167.stdout:2941: finishing write tid 2 to vpm1675861-29 2014-08-08T10:38:59.555 INFO:tasks.rados.rados.0.vpm167.stdout:2941: finishing write tid 4 to vpm1675861-29 2014-08-08T10:38:59.555 INFO:tasks.rados.rados.0.vpm167.stdout:2941: finishing write tid 5 to vpm1675861-29 2014-08-08T10:38:59.555 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 879 (ObjNum 1108 snap 391 seq_num 1108) dirty exists 2014-08-08T10:38:59.556 INFO:tasks.rados.rados.0.vpm167.stdout:2943: finishing write tid 1 to vpm1675861-6 2014-08-08T10:38:59.557 INFO:tasks.rados.rados.0.vpm167.stdout:2943: finishing write tid 2 to vpm1675861-6 2014-08-08T10:38:59.557 INFO:tasks.rados.rados.0.vpm167.stdout:2943: finishing write tid 3 to vpm1675861-6 2014-08-08T10:38:59.557 INFO:tasks.rados.rados.0.vpm167.stdout:2943: finishing write tid 5 to vpm1675861-6 2014-08-08T10:38:59.557 INFO:tasks.rados.rados.0.vpm167.stdout:2943: finishing write tid 6 to vpm1675861-6 2014-08-08T10:38:59.557 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 6 v 1616 (ObjNum 1109 snap 391 seq_num 1109) dirty exists 2014-08-08T10:38:59.558 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 1 to vpm1675861-35 2014-08-08T10:38:59.558 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 2 to vpm1675861-35 2014-08-08T10:38:59.558 INFO:tasks.rados.rados.0.vpm167.stdout:2941: done (2 left) 2014-08-08T10:38:59.558 INFO:tasks.rados.rados.0.vpm167.stdout:2943: done (1 left) 2014-08-08T10:38:59.558 INFO:tasks.rados.rados.0.vpm167.stdout:2946: read oid 40 snap -1 2014-08-08T10:38:59.559 INFO:tasks.rados.rados.0.vpm167.stdout:2947: snap_create 2014-08-08T10:38:59.587 INFO:tasks.rados.rados.0.vpm167.stdout:2946: expect (ObjNum 990 snap 335 seq_num 990) 2014-08-08T10:39:01.222 INFO:tasks.rados.rados.0.vpm167.stdout:2946: done (2 left) 2014-08-08T10:39:01.222 INFO:tasks.rados.rados.0.vpm167.stdout:2947: done (1 left) 2014-08-08T10:39:01.222 INFO:tasks.rados.rados.0.vpm167.stdout:2948: write oid 6 current snap is 392 2014-08-08T10:39:01.223 INFO:tasks.rados.rados.0.vpm167.stdout:2948: seq_num 1112 ranges {402479=483901,1516792=715910,2800332=414802,3602478=1} 2014-08-08T10:39:01.241 INFO:tasks.rados.rados.0.vpm167.stdout:2948: writing vpm1675861-6 from 402479 to 886380 tid 1 2014-08-08T10:39:01.294 INFO:tasks.rados.rados.0.vpm167.stdout:2948: writing vpm1675861-6 from 1516792 to 2232702 tid 2 2014-08-08T10:39:01.324 INFO:tasks.rados.rados.0.vpm167.stdout:2948: writing vpm1675861-6 from 2800332 to 3215134 tid 3 2014-08-08T10:39:01.329 INFO:tasks.rados.rados.0.vpm167.stdout:2948: writing vpm1675861-6 from 3602478 to 3602479 tid 4 2014-08-08T10:39:01.330 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 3 to vpm1675861-35 2014-08-08T10:39:01.330 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 4 to vpm1675861-35 2014-08-08T10:39:01.331 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 6 to vpm1675861-35 2014-08-08T10:39:01.331 INFO:tasks.rados.rados.0.vpm167.stdout:2945: finishing write tid 7 to vpm1675861-35 2014-08-08T10:39:01.331 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 35 v 888 (ObjNum 1111 snap 391 seq_num 1111) dirty exists 2014-08-08T10:39:01.332 INFO:tasks.rados.rados.0.vpm167.stdout:2945: done (1 left) 2014-08-08T10:39:01.332 INFO:tasks.rados.rados.0.vpm167.stdout:2949: snap_create 2014-08-08T10:39:01.802 INFO:tasks.rados.rados.0.vpm167.stdout:2949: done (1 left) 2014-08-08T10:39:01.802 INFO:tasks.rados.rados.0.vpm167.stdout:2950: read oid 24 snap -1 2014-08-08T10:39:01.803 INFO:tasks.rados.rados.0.vpm167.stdout:2951: snap_remove snap 357 2014-08-08T10:39:03.090 INFO:tasks.rados.rados.0.vpm167.stdout:2950: done (2 left) 2014-08-08T10:39:03.090 INFO:tasks.rados.rados.0.vpm167.stdout:2951: done (1 left) 2014-08-08T10:39:03.091 INFO:tasks.rados.rados.0.vpm167.stdout:2952: rollback oid 34 current snap is 393 2014-08-08T10:39:03.091 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 34 to 386 2014-08-08T10:39:03.092 INFO:tasks.rados.rados.0.vpm167.stdout:2953: snap_create 2014-08-08T10:39:04.163 INFO:tasks.rados.rados.0.vpm167.stdout:2953: done (2 left) 2014-08-08T10:39:04.164 INFO:tasks.rados.rados.0.vpm167.stdout:2954: snap_create 2014-08-08T10:39:04.965 INFO:tasks.ceph.osd.2:Stopped 2014-08-08T10:39:05.284 INFO:tasks.rados.rados.0.vpm167.stdout:2954: done (2 left) 2014-08-08T10:39:05.284 INFO:tasks.rados.rados.0.vpm167.stdout:2955: delete oid 24 current snap is 395 2014-08-08T10:39:05.290 INFO:tasks.rados.rados.0.vpm167.stdout:2955: done (2 left) 2014-08-08T10:39:05.290 INFO:tasks.rados.rados.0.vpm167.stdout:2956: write oid 27 current snap is 395 2014-08-08T10:39:05.290 INFO:tasks.rados.rados.0.vpm167.stdout:2956: seq_num 1114 ranges {408491=462581,1308960=299531} 2014-08-08T10:39:05.309 INFO:tasks.rados.rados.0.vpm167.stdout:2956: writing vpm1675861-27 from 408491 to 871072 tid 1 2014-08-08T10:39:05.340 INFO:tasks.rados.rados.0.vpm167.stdout:2956: writing vpm1675861-27 from 1308960 to 1608491 tid 2 2014-08-08T10:39:05.342 INFO:tasks.rados.rados.0.vpm167.stdout:2957: write oid 42 current snap is 395 2014-08-08T10:39:05.342 INFO:tasks.rados.rados.0.vpm167.stdout:2957: seq_num 1115 ranges {503721=586834,1303720=1} 2014-08-08T10:39:05.378 INFO:tasks.rados.rados.0.vpm167.stdout:2957: writing vpm1675861-42 from 503721 to 1090555 tid 1 2014-08-08T10:39:05.392 INFO:tasks.rados.rados.0.vpm167.stdout:2957: writing vpm1675861-42 from 1303720 to 1303721 tid 2 2014-08-08T10:39:05.393 INFO:tasks.rados.rados.0.vpm167.stdout:2956: finishing write tid 1 to vpm1675861-27 2014-08-08T10:39:05.394 INFO:tasks.rados.rados.0.vpm167.stdout:2956: finishing write tid 2 to vpm1675861-27 2014-08-08T10:39:05.395 INFO:tasks.rados.rados.0.vpm167.stdout:2956: finishing write tid 4 to vpm1675861-27 2014-08-08T10:39:05.395 INFO:tasks.rados.rados.0.vpm167.stdout:2958: read oid 30 snap -1 2014-08-08T10:39:05.395 INFO:tasks.rados.rados.0.vpm167.stdout:2959: write oid 13 current snap is 395 2014-08-08T10:39:05.395 INFO:tasks.rados.rados.0.vpm167.stdout:2959: seq_num 1116 ranges {448550=558638,1502152=499538,2048549=1} 2014-08-08T10:39:05.427 INFO:tasks.rados.rados.0.vpm167.stdout:2959: writing vpm1675861-13 from 448550 to 1007188 tid 1 2014-08-08T10:39:05.463 INFO:tasks.rados.rados.0.vpm167.stdout:2959: writing vpm1675861-13 from 1502152 to 2001690 tid 2 2014-08-08T10:39:05.468 INFO:tasks.rados.rados.0.vpm167.stdout:2959: writing vpm1675861-13 from 2048549 to 2048550 tid 3 2014-08-08T10:39:05.470 INFO:tasks.rados.rados.0.vpm167.stdout:2958: expect (ObjNum 1052 snap 362 seq_num 1052) 2014-08-08T10:39:06.182 INFO:tasks.rados.rados.0.vpm167.stdout:2958: done (5 left) 2014-08-08T10:39:06.183 INFO:tasks.rados.rados.0.vpm167.stdout:2960: read oid 20 snap -1 2014-08-08T10:39:06.183 INFO:tasks.rados.rados.0.vpm167.stdout:2961: read oid 9 snap -1 2014-08-08T10:39:06.185 INFO:tasks.rados.rados.0.vpm167.stdout:2962: snap_remove snap 383 2014-08-08T10:39:06.185 INFO:tasks.rados.rados.0.vpm167.stdout:2957: finishing write tid 1 to vpm1675861-42 2014-08-08T10:39:06.186 INFO:tasks.rados.rados.0.vpm167.stdout:2957: finishing write tid 2 to vpm1675861-42 2014-08-08T10:39:06.186 INFO:tasks.rados.rados.0.vpm167.stdout:2957: finishing write tid 4 to vpm1675861-42 2014-08-08T10:39:06.186 INFO:tasks.rados.rados.0.vpm167.stdout:2959: finishing write tid 1 to vpm1675861-13 2014-08-08T10:39:06.186 INFO:tasks.rados.rados.0.vpm167.stdout:2959: finishing write tid 2 to vpm1675861-13 2014-08-08T10:39:06.187 INFO:tasks.rados.rados.0.vpm167.stdout:2957: finishing write tid 5 to vpm1675861-42 2014-08-08T10:39:06.187 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 1841 (ObjNum 1115 snap 395 seq_num 1115) dirty exists 2014-08-08T10:39:06.187 INFO:tasks.rados.rados.0.vpm167.stdout:2959: finishing write tid 3 to vpm1675861-13 2014-08-08T10:39:06.187 INFO:tasks.rados.rados.0.vpm167.stdout:2959: finishing write tid 5 to vpm1675861-13 2014-08-08T10:39:06.188 INFO:tasks.rados.rados.0.vpm167.stdout:2956: finishing write tid 5 to vpm1675861-27 2014-08-08T10:39:06.189 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 27 v 2018 (ObjNum 1114 snap 395 seq_num 1114) dirty exists 2014-08-08T10:39:06.189 INFO:tasks.rados.rados.0.vpm167.stdout:2959: finishing write tid 6 to vpm1675861-13 2014-08-08T10:39:06.189 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1619 (ObjNum 1116 snap 395 seq_num 1116) dirty exists 2014-08-08T10:39:07.212 INFO:tasks.rados.rados.0.vpm167.stdout:2956: done (7 left) 2014-08-08T10:39:07.212 INFO:tasks.rados.rados.0.vpm167.stdout:2957: done (6 left) 2014-08-08T10:39:07.212 INFO:tasks.rados.rados.0.vpm167.stdout:2959: done (5 left) 2014-08-08T10:39:07.213 INFO:tasks.rados.rados.0.vpm167.stdout:2961: done (4 left) 2014-08-08T10:39:07.213 INFO:tasks.rados.rados.0.vpm167.stdout:2962: done (3 left) 2014-08-08T10:39:07.213 INFO:tasks.rados.rados.0.vpm167.stdout:2963: rollback oid 11 current snap is 395 2014-08-08T10:39:07.213 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 11 to 366 2014-08-08T10:39:07.214 INFO:tasks.rados.rados.0.vpm167.stdout:2964: snap_remove snap 374 2014-08-08T10:39:07.340 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 892 (ObjNum 1001 snap 342 seq_num 1001) dirty exists 2014-08-08T10:39:08.241 INFO:tasks.rados.rados.0.vpm167.stdout:2963: done (4 left) 2014-08-08T10:39:08.241 INFO:tasks.rados.rados.0.vpm167.stdout:2964: done (3 left) 2014-08-08T10:39:08.242 INFO:tasks.rados.rados.0.vpm167.stdout:2965: delete oid 45 current snap is 395 2014-08-08T10:39:08.758 INFO:tasks.rados.rados.0.vpm167.stdout:2965: done (3 left) 2014-08-08T10:39:08.758 INFO:tasks.rados.rados.0.vpm167.stdout:2966: snap_remove snap 367 2014-08-08T10:39:09.425 INFO:tasks.rados.rados.0.vpm167.stdout:2966: done (3 left) 2014-08-08T10:39:09.425 INFO:tasks.rados.rados.0.vpm167.stdout:2967: read oid 36 snap -1 2014-08-08T10:39:09.426 INFO:tasks.rados.rados.0.vpm167.stdout:2968: delete oid 32 current snap is 395 2014-08-08T10:39:09.430 INFO:tasks.rados.rados.0.vpm167.stdout:2968: done (4 left) 2014-08-08T10:39:09.430 INFO:tasks.rados.rados.0.vpm167.stdout:2969: snap_create 2014-08-08T10:39:09.965 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 0, 2, 4, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:09.965 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:09.965 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 5, 0, 2, 4, 3] 2014-08-08T10:39:09.966 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-08T10:39:10.766 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.0. 2014-08-08T10:39:10.910 INFO:tasks.rados.rados.0.vpm167.stdout:2967: done (4 left) 2014-08-08T10:39:10.910 INFO:tasks.rados.rados.0.vpm167.stdout:2969: done (3 left) 2014-08-08T10:39:10.911 INFO:tasks.rados.rados.0.vpm167.stdout:2970: write oid 11 current snap is 396 2014-08-08T10:39:10.911 INFO:tasks.rados.rados.0.vpm167.stdout:2970: seq_num 1119 ranges {700192=510147,1657029=745196,2700191=1} 2014-08-08T10:39:10.938 INFO:tasks.rados.rados.0.vpm167.stdout:2970: writing vpm1675861-11 from 700192 to 1210339 tid 1 2014-08-08T10:39:10.990 INFO:tasks.rados.rados.0.vpm167.stdout:2970: writing vpm1675861-11 from 1657029 to 2402225 tid 2 2014-08-08T10:39:10.996 INFO:tasks.rados.rados.0.vpm167.stdout:2970: writing vpm1675861-11 from 2700191 to 2700192 tid 3 2014-08-08T10:39:10.997 INFO:tasks.rados.rados.0.vpm167.stdout:2971: read oid 32 snap -1 2014-08-08T10:39:10.998 INFO:tasks.rados.rados.0.vpm167.stdout:2972: snap_remove snap 381 2014-08-08T10:39:11.882 INFO:tasks.rados.rados.0.vpm167.stdout:2971: done (5 left) 2014-08-08T10:39:11.883 INFO:tasks.rados.rados.0.vpm167.stdout:2972: done (4 left) 2014-08-08T10:39:11.883 INFO:tasks.rados.rados.0.vpm167.stdout:2973: read oid 10 snap 385 2014-08-08T10:39:11.883 INFO:tasks.rados.rados.0.vpm167.stdout:2974: rollback oid 49 current snap is 396 2014-08-08T10:39:11.884 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 49 to 368 2014-08-08T10:39:11.884 INFO:tasks.rados.rados.0.vpm167.stdout:2975: snap_remove snap 393 2014-08-08T10:39:12.241 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 49 v 1206 (ObjNum 955 snap 316 seq_num 955) dirty exists 2014-08-08T10:39:13.004 INFO:tasks.rados.rados.0.vpm167.stdout:2973: done (6 left) 2014-08-08T10:39:13.004 INFO:tasks.rados.rados.0.vpm167.stdout:2974: done (5 left) 2014-08-08T10:39:13.005 INFO:tasks.rados.rados.0.vpm167.stdout:2975: done (4 left) 2014-08-08T10:39:13.005 INFO:tasks.rados.rados.0.vpm167.stdout:2976: write oid 9 current snap is 396 2014-08-08T10:39:13.005 INFO:tasks.rados.rados.0.vpm167.stdout:2976: seq_num 1120 ranges {492572=1} 2014-08-08T10:39:13.010 INFO:tasks.rados.rados.0.vpm167.stdout:2976: writing vpm1675861-9 from 492572 to 492573 tid 1 2014-08-08T10:39:13.012 INFO:tasks.rados.rados.0.vpm167.stdout:2977: read oid 7 snap 376 2014-08-08T10:39:13.012 INFO:tasks.rados.rados.0.vpm167.stdout:2978: rollback oid 33 current snap is 396 2014-08-08T10:39:13.013 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 33 to 386 2014-08-08T10:39:13.013 INFO:tasks.rados.rados.0.vpm167.stdout:2979: rollback oid 26 current snap is 396 2014-08-08T10:39:13.013 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 26 to 375 2014-08-08T10:39:13.014 INFO:tasks.rados.rados.0.vpm167.stdout:2980: write oid 48 current snap is 396 2014-08-08T10:39:13.014 INFO:tasks.rados.rados.0.vpm167.stdout:2980: seq_num 1121 ranges {188826=1} 2014-08-08T10:39:13.016 INFO:tasks.rados.rados.0.vpm167.stdout:2980: writing vpm1675861-48 from 188826 to 188827 tid 1 2014-08-08T10:39:13.017 INFO:tasks.rados.rados.0.vpm167.stdout:2981: delete oid 40 current snap is 396 2014-08-08T10:39:13.046 INFO:tasks.rados.rados.0.vpm167.stdout:2977: expect (ObjNum 1083 snap 376 seq_num 1083) 2014-08-08T10:39:13.704 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 803 (ObjNum 1071 snap 371 seq_num 1071) dirty exists 2014-08-08T10:39:15.779 INFO:tasks.rados.rados.0.vpm167.stdout:2970: finishing write tid 1 to vpm1675861-11 2014-08-08T10:39:15.782 INFO:tasks.rados.rados.0.vpm167.stdout:2970: finishing write tid 2 to vpm1675861-11 2014-08-08T10:39:15.785 INFO:tasks.rados.rados.0.vpm167.stdout:2970: finishing write tid 3 to vpm1675861-11 2014-08-08T10:39:15.810 INFO:tasks.rados.rados.0.vpm167.stdout:2970: finishing write tid 5 to vpm1675861-11 2014-08-08T10:39:15.810 INFO:tasks.rados.rados.0.vpm167.stdout:2970: finishing write tid 6 to vpm1675861-11 2014-08-08T10:39:15.810 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 909 (ObjNum 1119 snap 396 seq_num 1119) dirty exists 2014-08-08T10:39:15.838 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4, 3] out_osds: [0] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:15.838 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:15.838 INFO:tasks.thrashosds.thrasher:Adding osd 0 2014-08-08T10:39:15.838 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 0' 2014-08-08T10:39:16.430 INFO:teuthology.orchestra.run.vpm166.stderr:marked in osd.0. 2014-08-08T10:39:16.469 INFO:tasks.thrashosds.thrasher:Added osd 0 2014-08-08T10:39:16.802 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1641 (ObjNum 889 snap 301 seq_num 889) dirty exists 2014-08-08T10:39:16.805 INFO:tasks.rados.rados.0.vpm167.stdout:2980: finishing write tid 1 to vpm1675861-48 2014-08-08T10:39:16.805 INFO:tasks.rados.rados.0.vpm167.stdout:2980: finishing write tid 3 to vpm1675861-48 2014-08-08T10:39:16.805 INFO:tasks.rados.rados.0.vpm167.stdout:2980: finishing write tid 4 to vpm1675861-48 2014-08-08T10:39:16.806 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 48 v 1644 (ObjNum 1121 snap 396 seq_num 1121) dirty exists 2014-08-08T10:39:21.469 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4, 3, 0] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:21.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:21.469 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 5, 2, 4, 3, 0] 2014-08-08T10:39:21.469 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-08-08T10:39:22.837 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.5. 2014-08-08T10:39:24.378 INFO:tasks.rados.rados.0.vpm167.stdout:2960: expect (ObjNum 1105 snap 391 seq_num 1105) 2014-08-08T10:39:24.589 INFO:tasks.rados.rados.0.vpm167.stdout:2960: done (9 left) 2014-08-08T10:39:24.590 INFO:tasks.rados.rados.0.vpm167.stdout:2970: done (8 left) 2014-08-08T10:39:24.590 INFO:tasks.rados.rados.0.vpm167.stdout:2977: done (7 left) 2014-08-08T10:39:24.590 INFO:tasks.rados.rados.0.vpm167.stdout:2978: done (6 left) 2014-08-08T10:39:24.590 INFO:tasks.rados.rados.0.vpm167.stdout:2979: done (5 left) 2014-08-08T10:39:24.591 INFO:tasks.rados.rados.0.vpm167.stdout:2980: done (4 left) 2014-08-08T10:39:24.591 INFO:tasks.rados.rados.0.vpm167.stdout:2981: done (3 left) 2014-08-08T10:39:24.591 INFO:tasks.rados.rados.0.vpm167.stdout:2982: rollback oid 37 current snap is 396 2014-08-08T10:39:24.591 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 37 to 344 2014-08-08T10:39:24.591 INFO:tasks.rados.rados.0.vpm167.stdout:2983: read oid 27 snap 385 2014-08-08T10:39:24.592 INFO:tasks.rados.rados.0.vpm167.stdout:2984: read oid 38 snap -1 2014-08-08T10:39:24.592 INFO:tasks.rados.rados.0.vpm167.stdout:2985: read oid 20 snap 372 2014-08-08T10:39:24.593 INFO:tasks.rados.rados.0.vpm167.stdout:2986: rollback oid 43 current snap is 396 2014-08-08T10:39:24.593 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 43 to 382 2014-08-08T10:39:24.593 INFO:tasks.rados.rados.0.vpm167.stdout:2987: read oid 30 snap 369 2014-08-08T10:39:24.594 INFO:tasks.rados.rados.0.vpm167.stdout:2988: write oid 36 current snap is 396 2014-08-08T10:39:24.594 INFO:tasks.rados.rados.0.vpm167.stdout:2988: seq_num 1123 ranges {593814=584885,1889941=303873} 2014-08-08T10:39:24.618 INFO:tasks.rados.rados.0.vpm167.stdout:2988: writing vpm1675861-36 from 593814 to 1178699 tid 1 2014-08-08T10:39:24.640 INFO:tasks.rados.rados.0.vpm167.stdout:2988: writing vpm1675861-36 from 1889941 to 2193814 tid 2 2014-08-08T10:39:24.641 INFO:tasks.rados.rados.0.vpm167.stdout:2985: done (9 left) 2014-08-08T10:39:24.641 INFO:tasks.rados.rados.0.vpm167.stdout:2989: write oid 47 current snap is 396 2014-08-08T10:39:24.641 INFO:tasks.rados.rados.0.vpm167.stdout:2989: seq_num 1124 ranges {690068=556689,1890067=1} 2014-08-08T10:39:24.666 INFO:tasks.rados.rados.0.vpm167.stdout:2989: writing vpm1675861-47 from 690068 to 1246757 tid 1 2014-08-08T10:39:24.677 INFO:tasks.rados.rados.0.vpm167.stdout:2989: writing vpm1675861-47 from 1890067 to 1890068 tid 2 2014-08-08T10:39:24.678 INFO:tasks.rados.rados.0.vpm167.stdout:2990: write oid 19 current snap is 396 2014-08-08T10:39:24.678 INFO:tasks.rados.rados.0.vpm167.stdout:2990: seq_num 1125 ranges {785298=680941,1585297=1} 2014-08-08T10:39:24.712 INFO:tasks.rados.rados.0.vpm167.stdout:2990: writing vpm1675861-19 from 785298 to 1466239 tid 1 2014-08-08T10:39:24.715 INFO:tasks.rados.rados.0.vpm167.stdout:2990: writing vpm1675861-19 from 1585297 to 1585298 tid 2 2014-08-08T10:39:24.715 INFO:tasks.rados.rados.0.vpm167.stdout:2987: expect (ObjNum 1052 snap 362 seq_num 1052) 2014-08-08T10:39:25.437 INFO:tasks.rados.rados.0.vpm167.stdout:2987: done (10 left) 2014-08-08T10:39:25.438 INFO:tasks.rados.rados.0.vpm167.stdout:2991: delete oid 1 current snap is 396 2014-08-08T10:39:25.438 INFO:tasks.rados.rados.0.vpm167.stdout:2989: finishing write tid 1 to vpm1675861-47 2014-08-08T10:39:25.439 INFO:tasks.rados.rados.0.vpm167.stdout:2989: finishing write tid 2 to vpm1675861-47 2014-08-08T10:39:25.440 INFO:tasks.rados.rados.0.vpm167.stdout:2989: finishing write tid 4 to vpm1675861-47 2014-08-08T10:39:25.441 INFO:tasks.rados.rados.0.vpm167.stdout:2990: finishing write tid 1 to vpm1675861-19 2014-08-08T10:39:25.441 INFO:tasks.rados.rados.0.vpm167.stdout:2990: finishing write tid 2 to vpm1675861-19 2014-08-08T10:39:25.441 INFO:tasks.rados.rados.0.vpm167.stdout:2989: finishing write tid 5 to vpm1675861-47 2014-08-08T10:39:25.442 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 47 v 1277 (ObjNum 1124 snap 396 seq_num 1124) dirty exists 2014-08-08T10:39:25.442 INFO:tasks.rados.rados.0.vpm167.stdout:2990: finishing write tid 4 to vpm1675861-19 2014-08-08T10:39:25.442 INFO:tasks.rados.rados.0.vpm167.stdout:2990: finishing write tid 5 to vpm1675861-19 2014-08-08T10:39:25.442 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 19 v 1280 (ObjNum 1125 snap 396 seq_num 1125) dirty exists 2014-08-08T10:39:25.443 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 1 to vpm1675861-6 2014-08-08T10:39:25.443 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 2 to vpm1675861-6 2014-08-08T10:39:25.443 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 3 to vpm1675861-6 2014-08-08T10:39:25.445 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 4 to vpm1675861-6 2014-08-08T10:39:25.445 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 6 to vpm1675861-6 2014-08-08T10:39:25.445 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1627 (ObjNum 30453184 snap 0 seq_num 40700480) dirty dne 2014-08-08T10:39:25.446 INFO:tasks.rados.rados.0.vpm167.stdout:2976: finishing write tid 1 to vpm1675861-9 2014-08-08T10:39:25.446 INFO:tasks.rados.rados.0.vpm167.stdout:2976: finishing write tid 3 to vpm1675861-9 2014-08-08T10:39:25.446 INFO:tasks.rados.rados.0.vpm167.stdout:2976: finishing write tid 4 to vpm1675861-9 2014-08-08T10:39:25.446 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 9 v 1630 (ObjNum 1120 snap 396 seq_num 1120) dirty exists 2014-08-08T10:39:25.447 INFO:tasks.rados.rados.0.vpm167.stdout:2948: finishing write tid 7 to vpm1675861-6 2014-08-08T10:39:25.447 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 6 v 1622 (ObjNum 1112 snap 392 seq_num 1112) dirty exists 2014-08-08T10:39:26.476 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 37 v 0 (ObjNum 320 snap 0 seq_num 4294967295) dirty dne 2014-08-08T10:39:26.963 INFO:tasks.rados.rados.0.vpm167.stdout:2988: finishing write tid 1 to vpm1675861-36 2014-08-08T10:39:26.963 INFO:tasks.rados.rados.0.vpm167.stdout:2988: finishing write tid 2 to vpm1675861-36 2014-08-08T10:39:26.997 INFO:tasks.rados.rados.0.vpm167.stdout:2988: finishing write tid 4 to vpm1675861-36 2014-08-08T10:39:26.997 INFO:tasks.rados.rados.0.vpm167.stdout:2988: finishing write tid 5 to vpm1675861-36 2014-08-08T10:39:26.998 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 36 v 1865 (ObjNum 1123 snap 396 seq_num 1123) dirty exists 2014-08-08T10:39:27.376 INFO:tasks.rados.rados.0.vpm167.stdout:2948: done (10 left) 2014-08-08T10:39:27.377 INFO:tasks.rados.rados.0.vpm167.stdout:2952: done (9 left) 2014-08-08T10:39:27.377 INFO:tasks.rados.rados.0.vpm167.stdout:2976: done (8 left) 2014-08-08T10:39:27.378 INFO:tasks.rados.rados.0.vpm167.stdout:2982: done (7 left) 2014-08-08T10:39:27.378 INFO:tasks.rados.rados.0.vpm167.stdout:2988: done (6 left) 2014-08-08T10:39:27.378 INFO:tasks.rados.rados.0.vpm167.stdout:2989: done (5 left) 2014-08-08T10:39:27.379 INFO:tasks.rados.rados.0.vpm167.stdout:2990: done (4 left) 2014-08-08T10:39:27.379 INFO:tasks.rados.rados.0.vpm167.stdout:2991: done (3 left) 2014-08-08T10:39:27.379 INFO:tasks.rados.rados.0.vpm167.stdout:2992: snap_remove snap 373 2014-08-08T10:39:27.650 INFO:tasks.rados.rados.0.vpm167.stdout:2984: expect (ObjNum 1099 snap 385 seq_num 1099) 2014-08-08T10:39:27.898 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0] out_osds: [5] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:27.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:27.898 INFO:tasks.thrashosds.thrasher:Adding osd 5 2014-08-08T10:39:27.898 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-08-08T10:39:28.427 INFO:tasks.rados.rados.0.vpm167.stdout:2984: done (3 left) 2014-08-08T10:39:28.428 INFO:tasks.rados.rados.0.vpm167.stdout:2992: done (2 left) 2014-08-08T10:39:28.428 INFO:tasks.rados.rados.0.vpm167.stdout:2993: write oid 8 current snap is 396 2014-08-08T10:39:28.428 INFO:tasks.rados.rados.0.vpm167.stdout:2993: seq_num 1127 ranges {789261=659622,1909349=444677,2945239=425203,3589260=1} 2014-08-08T10:39:28.473 INFO:tasks.rados.rados.0.vpm167.stdout:2993: writing vpm1675861-8 from 789261 to 1448883 tid 1 2014-08-08T10:39:28.528 INFO:tasks.rados.rados.0.vpm167.stdout:2993: writing vpm1675861-8 from 1909349 to 2354026 tid 2 2014-08-08T10:39:28.557 INFO:tasks.rados.rados.0.vpm167.stdout:2993: writing vpm1675861-8 from 2945239 to 3370442 tid 3 2014-08-08T10:39:28.561 INFO:tasks.rados.rados.0.vpm167.stdout:2993: writing vpm1675861-8 from 3589260 to 3589261 tid 4 2014-08-08T10:39:28.562 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 0 (ObjNum 30118816 snap 0 seq_num 0) dirty dne 2014-08-08T10:39:28.562 INFO:tasks.rados.rados.0.vpm167.stdout:2983: expect (ObjNum 1029 snap 353 seq_num 1029) 2014-08-08T10:39:29.406 INFO:teuthology.orchestra.run.vpm166.stderr:marked in osd.5. 2014-08-08T10:39:29.442 INFO:tasks.thrashosds.thrasher:Added osd 5 2014-08-08T10:39:31.400 INFO:tasks.rados.rados.0.vpm167.stdout:2983: done (2 left) 2014-08-08T10:39:31.402 INFO:tasks.rados.rados.0.vpm167.stdout:2986: done (1 left) 2014-08-08T10:39:31.402 INFO:tasks.rados.rados.0.vpm167.stdout:2994: snap_create 2014-08-08T10:39:31.402 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 1 to vpm1675861-8 2014-08-08T10:39:31.403 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 2 to vpm1675861-8 2014-08-08T10:39:31.403 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 3 to vpm1675861-8 2014-08-08T10:39:31.403 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 4 to vpm1675861-8 2014-08-08T10:39:31.404 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 6 to vpm1675861-8 2014-08-08T10:39:31.796 INFO:tasks.rados.rados.0.vpm167.stdout:2994: done (1 left) 2014-08-08T10:39:31.796 INFO:tasks.rados.rados.0.vpm167.stdout:2995: write oid 14 current snap is 397 2014-08-08T10:39:31.798 INFO:tasks.rados.rados.0.vpm167.stdout:2995: seq_num 1128 ranges {485515=782850,1738809=763449,2994017=291498} 2014-08-08T10:39:31.842 INFO:tasks.rados.rados.0.vpm167.stdout:2995: writing vpm1675861-14 from 485515 to 1268365 tid 1 2014-08-08T10:39:31.892 INFO:tasks.rados.rados.0.vpm167.stdout:2995: writing vpm1675861-14 from 1738809 to 2502258 tid 2 2014-08-08T10:39:31.916 INFO:tasks.rados.rados.0.vpm167.stdout:2995: writing vpm1675861-14 from 2994017 to 3285515 tid 3 2014-08-08T10:39:31.918 INFO:tasks.rados.rados.0.vpm167.stdout:2996: snap_create 2014-08-08T10:39:31.968 INFO:tasks.rados.rados.0.vpm167.stdout:2995: finishing write tid 1 to vpm1675861-14 2014-08-08T10:39:31.991 INFO:tasks.rados.rados.0.vpm167.stdout:2995: finishing write tid 2 to vpm1675861-14 2014-08-08T10:39:32.034 INFO:tasks.rados.rados.0.vpm167.stdout:2995: finishing write tid 3 to vpm1675861-14 2014-08-08T10:39:32.046 INFO:tasks.rados.rados.0.vpm167.stdout:2995: finishing write tid 5 to vpm1675861-14 2014-08-08T10:39:32.046 INFO:tasks.rados.rados.0.vpm167.stdout:2995: finishing write tid 6 to vpm1675861-14 2014-08-08T10:39:32.047 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1875 (ObjNum 1128 snap 397 seq_num 1128) dirty exists 2014-08-08T10:39:32.945 INFO:tasks.rados.rados.0.vpm167.stdout:2995: done (2 left) 2014-08-08T10:39:32.946 INFO:tasks.rados.rados.0.vpm167.stdout:2996: done (1 left) 2014-08-08T10:39:32.946 INFO:tasks.rados.rados.0.vpm167.stdout:2997: write oid 33 current snap is 398 2014-08-08T10:39:32.947 INFO:tasks.rados.rados.0.vpm167.stdout:2997: seq_num 1129 ranges {678921=400000} 2014-08-08T10:39:32.972 INFO:tasks.rados.rados.0.vpm167.stdout:2997: writing vpm1675861-33 from 678921 to 1078921 tid 1 2014-08-08T10:39:32.980 INFO:tasks.rados.rados.0.vpm167.stdout:2998: read oid 4 snap -1 2014-08-08T10:39:32.981 INFO:tasks.rados.rados.0.vpm167.stdout:2999: write oid 28 current snap is 398 2014-08-08T10:39:32.981 INFO:tasks.rados.rados.0.vpm167.stdout:2999: seq_num 1130 ranges {774150=1} 2014-08-08T10:39:32.988 INFO:tasks.rados.rados.0.vpm167.stdout:2999: writing vpm1675861-28 from 774150 to 774151 tid 1 2014-08-08T10:39:32.990 INFO:tasks.rados.rados.0.vpm167.stdout:2998: done (3 left) 2014-08-08T10:39:32.990 INFO:tasks.rados.rados.0.vpm167.stdout:3000: snap_create 2014-08-08T10:39:33.100 INFO:tasks.rados.rados.0.vpm167.stdout:2999: finishing write tid 1 to vpm1675861-28 2014-08-08T10:39:33.305 INFO:tasks.rados.rados.0.vpm167.stdout:2999: finishing write tid 3 to vpm1675861-28 2014-08-08T10:39:33.343 INFO:tasks.rados.rados.0.vpm167.stdout:2999: finishing write tid 4 to vpm1675861-28 2014-08-08T10:39:33.343 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 717 (ObjNum 1130 snap 398 seq_num 1130) dirty exists 2014-08-08T10:39:34.441 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:34.442 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:34.442 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2014-08-08T10:39:34.442 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:39:34.452 INFO:tasks.rados.rados.0.vpm167.stdout:2999: done (3 left) 2014-08-08T10:39:34.453 INFO:tasks.rados.rados.0.vpm167.stdout:3000: done (2 left) 2014-08-08T10:39:34.453 INFO:tasks.rados.rados.0.vpm167.stdout:3001: write oid 5 current snap is 399 2014-08-08T10:39:34.453 INFO:tasks.rados.rados.0.vpm167.stdout:3001: seq_num 1131 ranges {470403=1} 2014-08-08T10:39:34.459 INFO:tasks.rados.rados.0.vpm167.stdout:3001: writing vpm1675861-5 from 470403 to 470404 tid 1 2014-08-08T10:39:34.460 INFO:tasks.rados.rados.0.vpm167.stdout:3002: delete oid 41 current snap is 399 2014-08-08T10:39:34.611 INFO:tasks.rados.rados.0.vpm167.stdout:3001: finishing write tid 1 to vpm1675861-5 2014-08-08T10:39:34.612 INFO:tasks.rados.rados.0.vpm167.stdout:3001: finishing write tid 3 to vpm1675861-5 2014-08-08T10:39:34.613 INFO:tasks.rados.rados.0.vpm167.stdout:3001: finishing write tid 4 to vpm1675861-5 2014-08-08T10:39:34.613 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 5 v 1119 (ObjNum 1131 snap 399 seq_num 1131) dirty exists 2014-08-08T10:39:34.622 INFO:tasks.rados.rados.0.vpm167.stdout:2993: finishing write tid 7 to vpm1675861-8 2014-08-08T10:39:34.622 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 8 v 2034 (ObjNum 1127 snap 396 seq_num 1127) dirty exists 2014-08-08T10:39:34.641 INFO:tasks.rados.rados.0.vpm167.stdout:2997: finishing write tid 1 to vpm1675861-33 2014-08-08T10:39:34.641 INFO:tasks.rados.rados.0.vpm167.stdout:2997: finishing write tid 3 to vpm1675861-33 2014-08-08T10:39:34.694 INFO:tasks.rados.rados.0.vpm167.stdout:2997: finishing write tid 4 to vpm1675861-33 2014-08-08T10:39:34.694 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1651 (ObjNum 1129 snap 398 seq_num 1129) dirty exists 2014-08-08T10:39:35.064 INFO:tasks.rados.rados.0.vpm167.stdout:2993: done (3 left) 2014-08-08T10:39:35.064 INFO:tasks.rados.rados.0.vpm167.stdout:2997: done (2 left) 2014-08-08T10:39:35.064 INFO:tasks.rados.rados.0.vpm167.stdout:3001: done (1 left) 2014-08-08T10:39:35.064 INFO:tasks.rados.rados.0.vpm167.stdout:3002: done (0 left) 2014-08-08T10:39:35.065 INFO:tasks.rados.rados.0.vpm167.stdout:3003: snap_create 2014-08-08T10:39:35.227 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:39:35.288 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_1 pgp_num 36' 2014-08-08T10:39:35.696 INFO:tasks.rados.rados.0.vpm167.stdout:3003: done (0 left) 2014-08-08T10:39:35.702 INFO:tasks.rados.rados.0.vpm167.stdout:3004: write oid 12 current snap is 400 2014-08-08T10:39:35.702 INFO:tasks.rados.rados.0.vpm167.stdout:3004: seq_num 1133 ranges {475392=678992,1936479=538913} 2014-08-08T10:39:35.734 INFO:tasks.rados.rados.0.vpm167.stdout:3004: writing vpm1675861-12 from 475392 to 1154384 tid 1 2014-08-08T10:39:35.773 INFO:tasks.rados.rados.0.vpm167.stdout:3004: writing vpm1675861-12 from 1936479 to 2475392 tid 2 2014-08-08T10:39:35.775 INFO:tasks.rados.rados.0.vpm167.stdout:3004: finishing write tid 1 to vpm1675861-12 2014-08-08T10:39:35.775 INFO:tasks.rados.rados.0.vpm167.stdout:3005: rollback oid 38 current snap is 400 2014-08-08T10:39:35.776 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 38 to 379 2014-08-08T10:39:35.776 INFO:tasks.rados.rados.0.vpm167.stdout:3006: read oid 34 snap 369 2014-08-08T10:39:35.777 INFO:tasks.rados.rados.0.vpm167.stdout:3007: write oid 1 current snap is 400 2014-08-08T10:39:35.777 INFO:tasks.rados.rados.0.vpm167.stdout:3007: seq_num 1134 ranges {571646=685868,1934259=237387} 2014-08-08T10:39:35.816 INFO:tasks.rados.rados.0.vpm167.stdout:3007: writing vpm1675861-1 from 571646 to 1257514 tid 1 2014-08-08T10:39:35.842 INFO:tasks.rados.rados.0.vpm167.stdout:3007: writing vpm1675861-1 from 1934259 to 2171646 tid 2 2014-08-08T10:39:35.844 INFO:tasks.rados.rados.0.vpm167.stdout:3006: done (3 left) 2014-08-08T10:39:35.844 INFO:tasks.rados.rados.0.vpm167.stdout:3008: rollback oid 23 current snap is 400 2014-08-08T10:39:35.845 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 23 to 379 2014-08-08T10:39:35.845 INFO:tasks.rados.rados.0.vpm167.stdout:3009: write oid 3 current snap is 400 2014-08-08T10:39:35.846 INFO:tasks.rados.rados.0.vpm167.stdout:3009: seq_num 1135 ranges {514427=657673,1611271=791711,2914426=1} 2014-08-08T10:39:35.872 INFO:tasks.rados.rados.0.vpm167.stdout:3009: writing vpm1675861-3 from 514427 to 1172100 tid 1 2014-08-08T10:39:35.907 INFO:tasks.rados.rados.0.vpm167.stdout:3009: writing vpm1675861-3 from 1611271 to 2402982 tid 2 2014-08-08T10:39:35.913 INFO:tasks.rados.rados.0.vpm167.stdout:3009: writing vpm1675861-3 from 2914426 to 2914427 tid 3 2014-08-08T10:39:35.914 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 23 v 811 (ObjNum 1078 snap 374 seq_num 1078) dirty exists 2014-08-08T10:39:35.915 INFO:tasks.rados.rados.0.vpm167.stdout:3008: done (4 left) 2014-08-08T10:39:35.915 INFO:tasks.rados.rados.0.vpm167.stdout:3010: snap_remove snap 391 2014-08-08T10:39:35.927 INFO:tasks.rados.rados.0.vpm167.stdout:3009: finishing write tid 1 to vpm1675861-3 2014-08-08T10:39:35.963 INFO:tasks.rados.rados.0.vpm167.stdout:3009: finishing write tid 2 to vpm1675861-3 2014-08-08T10:39:35.968 INFO:tasks.rados.rados.0.vpm167.stdout:3009: finishing write tid 3 to vpm1675861-3 2014-08-08T10:39:35.974 INFO:tasks.rados.rados.0.vpm167.stdout:3009: finishing write tid 5 to vpm1675861-3 2014-08-08T10:39:35.974 INFO:tasks.rados.rados.0.vpm167.stdout:3009: finishing write tid 6 to vpm1675861-3 2014-08-08T10:39:35.975 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 710 (ObjNum 1135 snap 400 seq_num 1135) dirty exists 2014-08-08T10:39:36.034 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 38 v 786 (ObjNum 1027 snap 351 seq_num 1027) dirty exists 2014-08-08T10:39:36.042 INFO:tasks.rados.rados.0.vpm167.stdout:3007: finishing write tid 1 to vpm1675861-1 2014-08-08T10:39:36.042 INFO:tasks.rados.rados.0.vpm167.stdout:3007: finishing write tid 2 to vpm1675861-1 2014-08-08T10:39:36.043 INFO:tasks.rados.rados.0.vpm167.stdout:3007: finishing write tid 4 to vpm1675861-1 2014-08-08T10:39:36.044 INFO:tasks.rados.rados.0.vpm167.stdout:3004: finishing write tid 2 to vpm1675861-12 2014-08-08T10:39:36.044 INFO:tasks.rados.rados.0.vpm167.stdout:3004: finishing write tid 4 to vpm1675861-12 2014-08-08T10:39:36.047 INFO:tasks.rados.rados.0.vpm167.stdout:3004: finishing write tid 5 to vpm1675861-12 2014-08-08T10:39:36.047 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2040 (ObjNum 1133 snap 400 seq_num 1133) dirty exists 2014-08-08T10:39:36.405 INFO:tasks.rados.rados.0.vpm167.stdout:3007: finishing write tid 5 to vpm1675861-1 2014-08-08T10:39:36.405 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 1 v 1289 (ObjNum 1134 snap 400 seq_num 1134) dirty exists 2014-08-08T10:39:36.702 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 4 pgp_num to 36 2014-08-08T10:39:36.825 INFO:tasks.rados.rados.0.vpm167.stdout:3004: done (4 left) 2014-08-08T10:39:36.825 INFO:tasks.rados.rados.0.vpm167.stdout:3005: done (3 left) 2014-08-08T10:39:36.826 INFO:tasks.rados.rados.0.vpm167.stdout:3007: done (2 left) 2014-08-08T10:39:36.826 INFO:tasks.rados.rados.0.vpm167.stdout:3009: done (1 left) 2014-08-08T10:39:36.826 INFO:tasks.rados.rados.0.vpm167.stdout:3010: done (0 left) 2014-08-08T10:39:36.827 INFO:tasks.rados.rados.0.vpm167.stdout:3011: delete oid 42 current snap is 400 2014-08-08T10:39:37.071 INFO:tasks.rados.rados.0.vpm167.stdout:3011: done (0 left) 2014-08-08T10:39:37.071 INFO:tasks.rados.rados.0.vpm167.stdout:3012: rollback oid 24 current snap is 400 2014-08-08T10:39:37.071 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 24 to 390 2014-08-08T10:39:37.072 INFO:tasks.rados.rados.0.vpm167.stdout:3013: write oid 29 current snap is 400 2014-08-08T10:39:37.072 INFO:tasks.rados.rados.0.vpm167.stdout:3013: seq_num 1137 ranges {670839=754753,2204463=683525,3682998=187841} 2014-08-08T10:39:37.102 INFO:tasks.rados.rados.0.vpm167.stdout:3013: writing vpm1675861-29 from 670839 to 1425592 tid 1 2014-08-08T10:39:37.132 INFO:tasks.rados.rados.0.vpm167.stdout:3013: writing vpm1675861-29 from 2204463 to 2887988 tid 2 2014-08-08T10:39:37.146 INFO:tasks.rados.rados.0.vpm167.stdout:3013: writing vpm1675861-29 from 3682998 to 3870839 tid 3 2014-08-08T10:39:37.147 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 24 v 0 (ObjNum -1 snap 0 seq_num 0) dirty dne 2014-08-08T10:39:37.147 INFO:tasks.rados.rados.0.vpm167.stdout:3012: done (1 left) 2014-08-08T10:39:37.147 INFO:tasks.rados.rados.0.vpm167.stdout:3014: write oid 12 current snap is 400 2014-08-08T10:39:37.148 INFO:tasks.rados.rados.0.vpm167.stdout:3014: seq_num 1138 ranges {615668=1} 2014-08-08T10:39:37.152 INFO:tasks.rados.rados.0.vpm167.stdout:3014: writing vpm1675861-12 from 615668 to 615669 tid 1 2014-08-08T10:39:37.153 INFO:tasks.rados.rados.0.vpm167.stdout:3015: read oid 19 snap -1 2014-08-08T10:39:37.154 INFO:tasks.rados.rados.0.vpm167.stdout:3016: snap_remove snap 380 2014-08-08T10:39:37.165 INFO:tasks.rados.rados.0.vpm167.stdout:3013: finishing write tid 1 to vpm1675861-29 2014-08-08T10:39:37.179 INFO:tasks.rados.rados.0.vpm167.stdout:3014: finishing write tid 1 to vpm1675861-12 2014-08-08T10:39:37.182 INFO:tasks.rados.rados.0.vpm167.stdout:3013: finishing write tid 2 to vpm1675861-29 2014-08-08T10:39:37.186 INFO:tasks.rados.rados.0.vpm167.stdout:3014: finishing write tid 3 to vpm1675861-12 2014-08-08T10:39:37.186 INFO:tasks.rados.rados.0.vpm167.stdout:3014: finishing write tid 4 to vpm1675861-12 2014-08-08T10:39:37.186 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2048 (ObjNum 1138 snap 400 seq_num 1138) dirty exists 2014-08-08T10:39:37.187 INFO:tasks.rados.rados.0.vpm167.stdout:3015: expect (ObjNum 1125 snap 396 seq_num 1125) 2014-08-08T10:39:37.594 INFO:tasks.rados.rados.0.vpm167.stdout:3013: finishing write tid 3 to vpm1675861-29 2014-08-08T10:39:37.595 INFO:tasks.rados.rados.0.vpm167.stdout:3013: finishing write tid 5 to vpm1675861-29 2014-08-08T10:39:37.595 INFO:tasks.rados.rados.0.vpm167.stdout:3013: finishing write tid 6 to vpm1675861-29 2014-08-08T10:39:37.595 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 928 (ObjNum 1137 snap 400 seq_num 1137) dirty exists 2014-08-08T10:39:37.971 INFO:tasks.rados.rados.0.vpm167.stdout:3013: done (3 left) 2014-08-08T10:39:37.972 INFO:tasks.rados.rados.0.vpm167.stdout:3014: done (2 left) 2014-08-08T10:39:37.972 INFO:tasks.rados.rados.0.vpm167.stdout:3015: done (1 left) 2014-08-08T10:39:37.972 INFO:tasks.rados.rados.0.vpm167.stdout:3016: done (0 left) 2014-08-08T10:39:37.973 INFO:tasks.rados.rados.0.vpm167.stdout:3017: write oid 12 current snap is 400 2014-08-08T10:39:37.973 INFO:tasks.rados.rados.0.vpm167.stdout:3017: seq_num 1139 ranges {560498=733434,1360497=1} 2014-08-08T10:39:38.004 INFO:tasks.rados.rados.0.vpm167.stdout:3017: writing vpm1675861-12 from 560498 to 1293932 tid 1 2014-08-08T10:39:38.008 INFO:tasks.rados.rados.0.vpm167.stdout:3017: writing vpm1675861-12 from 1360497 to 1360498 tid 2 2014-08-08T10:39:38.008 INFO:tasks.rados.rados.0.vpm167.stdout:3018: snap_create 2014-08-08T10:39:38.179 INFO:tasks.rados.rados.0.vpm167.stdout:3017: finishing write tid 1 to vpm1675861-12 2014-08-08T10:39:38.179 INFO:tasks.rados.rados.0.vpm167.stdout:3017: finishing write tid 2 to vpm1675861-12 2014-08-08T10:39:38.180 INFO:tasks.rados.rados.0.vpm167.stdout:3017: finishing write tid 4 to vpm1675861-12 2014-08-08T10:39:38.180 INFO:tasks.rados.rados.0.vpm167.stdout:3017: finishing write tid 5 to vpm1675861-12 2014-08-08T10:39:38.181 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2055 (ObjNum 1139 snap 400 seq_num 1139) dirty exists 2014-08-08T10:39:39.516 INFO:tasks.rados.rados.0.vpm167.stdout:3017: done (1 left) 2014-08-08T10:39:39.516 INFO:tasks.rados.rados.0.vpm167.stdout:3018: done (0 left) 2014-08-08T10:39:39.517 INFO:tasks.rados.rados.0.vpm167.stdout:3019: rollback oid 14 current snap is 401 2014-08-08T10:39:39.517 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 358 2014-08-08T10:39:39.517 INFO:tasks.rados.rados.0.vpm167.stdout:3020: snap_create 2014-08-08T10:39:39.634 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1887 (ObjNum 30780080 snap 0 seq_num 32) dirty dne 2014-08-08T10:39:40.504 INFO:tasks.rados.rados.0.vpm167.stdout:3019: done (1 left) 2014-08-08T10:39:40.504 INFO:tasks.rados.rados.0.vpm167.stdout:3020: done (0 left) 2014-08-08T10:39:40.505 INFO:tasks.rados.rados.0.vpm167.stdout:3021: rollback oid 41 current snap is 402 2014-08-08T10:39:40.505 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 41 to 368 2014-08-08T10:39:40.505 INFO:tasks.rados.rados.0.vpm167.stdout:3022: snap_remove snap 401 2014-08-08T10:39:40.677 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 0 (ObjNum 31573912 snap 0 seq_num 39699600) dirty dne 2014-08-08T10:39:41.511 INFO:tasks.rados.rados.0.vpm167.stdout:3021: done (1 left) 2014-08-08T10:39:41.512 INFO:tasks.rados.rados.0.vpm167.stdout:3022: done (0 left) 2014-08-08T10:39:41.512 INFO:tasks.rados.rados.0.vpm167.stdout:3023: read oid 20 snap -1 2014-08-08T10:39:41.512 INFO:tasks.rados.rados.0.vpm167.stdout:3024: snap_create 2014-08-08T10:39:41.563 INFO:tasks.rados.rados.0.vpm167.stdout:3023: expect (ObjNum 1105 snap 391 seq_num 1105) 2014-08-08T10:39:41.761 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-08-08T10:39:41.761 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:41.761 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 0, 5, 4, 3] 2014-08-08T10:39:42.567 INFO:tasks.rados.rados.0.vpm167.stdout:3023: done (1 left) 2014-08-08T10:39:42.568 INFO:tasks.rados.rados.0.vpm167.stdout:3024: done (0 left) 2014-08-08T10:39:42.568 INFO:tasks.rados.rados.0.vpm167.stdout:3025: read oid 45 snap 394 2014-08-08T10:39:42.568 INFO:tasks.rados.rados.0.vpm167.stdout:3026: rollback oid 28 current snap is 403 2014-08-08T10:39:42.568 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 28 to 392 2014-08-08T10:39:42.569 INFO:tasks.rados.rados.0.vpm167.stdout:3027: write oid 43 current snap is 403 2014-08-08T10:39:42.569 INFO:tasks.rados.rados.0.vpm167.stdout:3027: seq_num 1140 ranges {504304=706262,1808939=295365} 2014-08-08T10:39:42.610 INFO:tasks.rados.rados.0.vpm167.stdout:3027: writing vpm1675861-43 from 504304 to 1210566 tid 1 2014-08-08T10:39:42.631 INFO:tasks.rados.rados.0.vpm167.stdout:3027: writing vpm1675861-43 from 1808939 to 2104304 tid 2 2014-08-08T10:39:42.632 INFO:tasks.rados.rados.0.vpm167.stdout:3028: read oid 46 snap -1 2014-08-08T10:39:42.633 INFO:tasks.rados.rados.0.vpm167.stdout:3029: read oid 20 snap -1 2014-08-08T10:39:42.634 INFO:tasks.rados.rados.0.vpm167.stdout:3030: snap_create 2014-08-08T10:39:42.647 INFO:tasks.rados.rados.0.vpm167.stdout:3029: expect (ObjNum 1105 snap 391 seq_num 1105) 2014-08-08T10:39:42.832 INFO:tasks.rados.rados.0.vpm167.stdout:3028: expect (ObjNum 1060 snap 364 seq_num 1060) 2014-08-08T10:39:43.681 INFO:tasks.rados.rados.0.vpm167.stdout:3028: done (5 left) 2014-08-08T10:39:43.681 INFO:tasks.rados.rados.0.vpm167.stdout:3029: done (4 left) 2014-08-08T10:39:43.681 INFO:tasks.rados.rados.0.vpm167.stdout:3030: done (3 left) 2014-08-08T10:39:43.682 INFO:tasks.rados.rados.0.vpm167.stdout:3031: write oid 11 current snap is 404 2014-08-08T10:39:43.682 INFO:tasks.rados.rados.0.vpm167.stdout:3031: seq_num 1141 ranges {716910=678067,2039424=277486} 2014-08-08T10:39:43.710 INFO:tasks.rados.rados.0.vpm167.stdout:3031: writing vpm1675861-11 from 716910 to 1394977 tid 1 2014-08-08T10:39:43.730 INFO:tasks.rados.rados.0.vpm167.stdout:3031: writing vpm1675861-11 from 2039424 to 2316910 tid 2 2014-08-08T10:39:43.731 INFO:tasks.rados.rados.0.vpm167.stdout:3032: delete oid 16 current snap is 404 2014-08-08T10:39:43.809 INFO:tasks.rados.rados.0.vpm167.stdout:3031: finishing write tid 1 to vpm1675861-11 2014-08-08T10:39:43.849 INFO:tasks.rados.rados.0.vpm167.stdout:3031: finishing write tid 2 to vpm1675861-11 2014-08-08T10:39:43.854 INFO:tasks.rados.rados.0.vpm167.stdout:3031: finishing write tid 4 to vpm1675861-11 2014-08-08T10:39:43.893 INFO:tasks.rados.rados.0.vpm167.stdout:3031: finishing write tid 5 to vpm1675861-11 2014-08-08T10:39:43.893 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 938 (ObjNum 1141 snap 404 seq_num 1141) dirty exists 2014-08-08T10:39:47.762 INFO:tasks.ceph.osd.3:Stopped 2014-08-08T10:39:52.763 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:39:52.764 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:52.764 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.382571526953 2014-08-08T10:39:52.764 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 0.382571526953' 2014-08-08T10:39:53.366 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.5 to 0.382572 (8250722) 2014-08-08T10:39:58.395 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:39:58.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:39:58.395 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2014-08-08T10:39:58.395 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:39:58.396 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:39:58.396 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-08-08T10:40:03.497 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:03.497 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:03.497 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.871385 2014-08-08T10:40:03.497 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 2 0.871384565525' 2014-08-08T10:40:03.814 INFO:tasks.rados.rados.0.vpm167.stdout:3027: finishing write tid 1 to vpm1675861-43 2014-08-08T10:40:03.816 INFO:tasks.rados.rados.0.vpm167.stdout:3027: finishing write tid 2 to vpm1675861-43 2014-08-08T10:40:03.817 INFO:tasks.rados.rados.0.vpm167.stdout:3027: finishing write tid 4 to vpm1675861-43 2014-08-08T10:40:03.819 INFO:tasks.rados.rados.0.vpm167.stdout:3027: finishing write tid 5 to vpm1675861-43 2014-08-08T10:40:03.819 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 793 (ObjNum 1140 snap 403 seq_num 1140) dirty exists 2014-08-08T10:40:03.936 INFO:tasks.rados.rados.0.vpm167.stdout:3025: expect (ObjNum 1080 snap 374 seq_num 1080) 2014-08-08T10:40:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3025: done (4 left) 2014-08-08T10:40:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3027: done (3 left) 2014-08-08T10:40:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3031: done (2 left) 2014-08-08T10:40:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3032: done (1 left) 2014-08-08T10:40:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3033: snap_create 2014-08-08T10:40:04.143 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 720 (ObjNum 30004608 snap 0 seq_num 0) dirty dne 2014-08-08T10:40:04.798 INFO:teuthology.orchestra.run.vpm166.stderr:set osd.2 primary-affinity to 0.871385 (8571072) 2014-08-08T10:40:04.806 INFO:tasks.rados.rados.0.vpm167.stdout:3026: done (1 left) 2014-08-08T10:40:04.806 INFO:tasks.rados.rados.0.vpm167.stdout:3033: done (0 left) 2014-08-08T10:40:04.806 INFO:tasks.rados.rados.0.vpm167.stdout:3034: read oid 47 snap -1 2014-08-08T10:40:04.807 INFO:tasks.rados.rados.0.vpm167.stdout:3035: rollback oid 20 current snap is 405 2014-08-08T10:40:04.807 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 20 to 358 2014-08-08T10:40:04.807 INFO:tasks.rados.rados.0.vpm167.stdout:3036: read oid 40 snap -1 2014-08-08T10:40:04.807 INFO:tasks.rados.rados.0.vpm167.stdout:3037: delete oid 4 current snap is 405 2014-08-08T10:40:04.817 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 20 v 879 (ObjNum 1041 snap 358 seq_num 1041) dirty exists 2014-08-08T10:40:04.840 INFO:tasks.rados.rados.0.vpm167.stdout:3035: done (3 left) 2014-08-08T10:40:04.841 INFO:tasks.rados.rados.0.vpm167.stdout:3037: done (2 left) 2014-08-08T10:40:04.841 INFO:tasks.rados.rados.0.vpm167.stdout:3038: rollback oid 7 current snap is 405 2014-08-08T10:40:04.841 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 7 to 388 2014-08-08T10:40:04.841 INFO:tasks.rados.rados.0.vpm167.stdout:3039: snap_remove snap 395 2014-08-08T10:40:04.853 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 712 (ObjNum 1083 snap 376 seq_num 1083) dirty exists 2014-08-08T10:40:04.879 INFO:tasks.rados.rados.0.vpm167.stdout:3034: expect (ObjNum 1124 snap 396 seq_num 1124) 2014-08-08T10:40:05.874 INFO:tasks.rados.rados.0.vpm167.stdout:3034: done (3 left) 2014-08-08T10:40:05.875 INFO:tasks.rados.rados.0.vpm167.stdout:3036: done (2 left) 2014-08-08T10:40:05.875 INFO:tasks.rados.rados.0.vpm167.stdout:3038: done (1 left) 2014-08-08T10:40:05.875 INFO:tasks.rados.rados.0.vpm167.stdout:3039: done (0 left) 2014-08-08T10:40:05.875 INFO:tasks.rados.rados.0.vpm167.stdout:3040: write oid 18 current snap is 405 2014-08-08T10:40:05.876 INFO:tasks.rados.rados.0.vpm167.stdout:3040: seq_num 1144 ranges {700775=781000,2228348=712812,3473400=27375} 2014-08-08T10:40:05.905 INFO:tasks.rados.rados.0.vpm167.stdout:3040: writing vpm1675861-18 from 700775 to 1481775 tid 1 2014-08-08T10:40:05.961 INFO:tasks.rados.rados.0.vpm167.stdout:3040: writing vpm1675861-18 from 2228348 to 2941160 tid 2 2014-08-08T10:40:05.970 INFO:tasks.rados.rados.0.vpm167.stdout:3040: writing vpm1675861-18 from 3473400 to 3500775 tid 3 2014-08-08T10:40:05.971 INFO:tasks.rados.rados.0.vpm167.stdout:3041: snap_remove snap 394 2014-08-08T10:40:06.032 INFO:tasks.rados.rados.0.vpm167.stdout:3040: finishing write tid 1 to vpm1675861-18 2014-08-08T10:40:06.156 INFO:tasks.rados.rados.0.vpm167.stdout:3040: finishing write tid 2 to vpm1675861-18 2014-08-08T10:40:06.157 INFO:tasks.rados.rados.0.vpm167.stdout:3040: finishing write tid 3 to vpm1675861-18 2014-08-08T10:40:06.158 INFO:tasks.rados.rados.0.vpm167.stdout:3040: finishing write tid 5 to vpm1675861-18 2014-08-08T10:40:06.159 INFO:tasks.rados.rados.0.vpm167.stdout:3040: finishing write tid 6 to vpm1675861-18 2014-08-08T10:40:06.159 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 18 v 1894 (ObjNum 1144 snap 405 seq_num 1144) dirty exists 2014-08-08T10:40:06.974 INFO:tasks.rados.rados.0.vpm167.stdout:3040: done (1 left) 2014-08-08T10:40:06.975 INFO:tasks.rados.rados.0.vpm167.stdout:3041: done (0 left) 2014-08-08T10:40:06.975 INFO:tasks.rados.rados.0.vpm167.stdout:3042: write oid 4 current snap is 405 2014-08-08T10:40:06.975 INFO:tasks.rados.rados.0.vpm167.stdout:3042: seq_num 1145 ranges {796005=752804,2057808=631583,3196004=1} 2014-08-08T10:40:07.005 INFO:tasks.rados.rados.0.vpm167.stdout:3042: writing vpm1675861-4 from 796005 to 1548809 tid 1 2014-08-08T10:40:07.031 INFO:tasks.rados.rados.0.vpm167.stdout:3042: writing vpm1675861-4 from 2057808 to 2689391 tid 2 2014-08-08T10:40:07.036 INFO:tasks.rados.rados.0.vpm167.stdout:3042: writing vpm1675861-4 from 3196004 to 3196005 tid 3 2014-08-08T10:40:07.037 INFO:tasks.rados.rados.0.vpm167.stdout:3043: delete oid 28 current snap is 405 2014-08-08T10:40:07.043 INFO:tasks.rados.rados.0.vpm167.stdout:3043: done (1 left) 2014-08-08T10:40:07.043 INFO:tasks.rados.rados.0.vpm167.stdout:3044: read oid 21 snap -1 2014-08-08T10:40:07.044 INFO:tasks.rados.rados.0.vpm167.stdout:3045: write oid 14 current snap is 405 2014-08-08T10:40:07.044 INFO:tasks.rados.rados.0.vpm167.stdout:3045: seq_num 1147 ranges {400992=732509,1200991=1} 2014-08-08T10:40:07.071 INFO:tasks.rados.rados.0.vpm167.stdout:3045: writing vpm1675861-14 from 400992 to 1133501 tid 1 2014-08-08T10:40:07.073 INFO:tasks.rados.rados.0.vpm167.stdout:3045: writing vpm1675861-14 from 1200991 to 1200992 tid 2 2014-08-08T10:40:07.074 INFO:tasks.rados.rados.0.vpm167.stdout:3046: delete oid 15 current snap is 405 2014-08-08T10:40:07.119 INFO:tasks.rados.rados.0.vpm167.stdout:3044: expect (ObjNum 1010 snap 345 seq_num 1010) 2014-08-08T10:40:08.083 INFO:tasks.rados.rados.0.vpm167.stdout:3044: done (3 left) 2014-08-08T10:40:08.083 INFO:tasks.rados.rados.0.vpm167.stdout:3046: done (2 left) 2014-08-08T10:40:08.084 INFO:tasks.rados.rados.0.vpm167.stdout:3047: write oid 8 current snap is 405 2014-08-08T10:40:08.084 INFO:tasks.rados.rados.0.vpm167.stdout:3047: seq_num 1149 ranges {690652=427541,1873569=415212,2690651=1} 2014-08-08T10:40:08.112 INFO:tasks.rados.rados.0.vpm167.stdout:3047: writing vpm1675861-8 from 690652 to 1118193 tid 1 2014-08-08T10:40:08.142 INFO:tasks.rados.rados.0.vpm167.stdout:3047: writing vpm1675861-8 from 1873569 to 2288781 tid 2 2014-08-08T10:40:08.148 INFO:tasks.rados.rados.0.vpm167.stdout:3047: writing vpm1675861-8 from 2690651 to 2690652 tid 3 2014-08-08T10:40:08.149 INFO:tasks.rados.rados.0.vpm167.stdout:3042: finishing write tid 1 to vpm1675861-4 2014-08-08T10:40:08.149 INFO:tasks.rados.rados.0.vpm167.stdout:3042: finishing write tid 2 to vpm1675861-4 2014-08-08T10:40:08.149 INFO:tasks.rados.rados.0.vpm167.stdout:3042: finishing write tid 3 to vpm1675861-4 2014-08-08T10:40:08.150 INFO:tasks.rados.rados.0.vpm167.stdout:3042: finishing write tid 5 to vpm1675861-4 2014-08-08T10:40:08.150 INFO:tasks.rados.rados.0.vpm167.stdout:3042: finishing write tid 6 to vpm1675861-4 2014-08-08T10:40:08.150 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 4 v 1128 (ObjNum 1145 snap 405 seq_num 1145) dirty exists 2014-08-08T10:40:08.150 INFO:tasks.rados.rados.0.vpm167.stdout:3045: finishing write tid 1 to vpm1675861-14 2014-08-08T10:40:08.151 INFO:tasks.rados.rados.0.vpm167.stdout:3045: finishing write tid 2 to vpm1675861-14 2014-08-08T10:40:08.151 INFO:tasks.rados.rados.0.vpm167.stdout:3045: finishing write tid 4 to vpm1675861-14 2014-08-08T10:40:08.151 INFO:tasks.rados.rados.0.vpm167.stdout:3045: finishing write tid 5 to vpm1675861-14 2014-08-08T10:40:08.152 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1902 (ObjNum 1147 snap 405 seq_num 1147) dirty exists 2014-08-08T10:40:08.152 INFO:tasks.rados.rados.0.vpm167.stdout:3047: finishing write tid 1 to vpm1675861-8 2014-08-08T10:40:08.152 INFO:tasks.rados.rados.0.vpm167.stdout:3042: done (2 left) 2014-08-08T10:40:08.153 INFO:tasks.rados.rados.0.vpm167.stdout:3045: done (1 left) 2014-08-08T10:40:08.153 INFO:tasks.rados.rados.0.vpm167.stdout:3048: snap_create 2014-08-08T10:40:08.155 INFO:tasks.rados.rados.0.vpm167.stdout:3047: finishing write tid 2 to vpm1675861-8 2014-08-08T10:40:08.159 INFO:tasks.rados.rados.0.vpm167.stdout:3047: finishing write tid 3 to vpm1675861-8 2014-08-08T10:40:08.160 INFO:tasks.rados.rados.0.vpm167.stdout:3047: finishing write tid 5 to vpm1675861-8 2014-08-08T10:40:08.161 INFO:tasks.rados.rados.0.vpm167.stdout:3047: finishing write tid 6 to vpm1675861-8 2014-08-08T10:40:08.161 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 8 v 2062 (ObjNum 1149 snap 405 seq_num 1149) dirty exists 2014-08-08T10:40:09.815 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 3, 0, 5] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:09.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:09.815 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 4, 3, 0, 5] 2014-08-08T10:40:09.816 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-08-08T10:40:10.197 INFO:tasks.rados.rados.0.vpm167.stdout:3047: done (1 left) 2014-08-08T10:40:10.202 INFO:tasks.rados.rados.0.vpm167.stdout:3048: done (0 left) 2014-08-08T10:40:10.202 INFO:tasks.rados.rados.0.vpm167.stdout:3049: write oid 12 current snap is 406 2014-08-08T10:40:10.202 INFO:tasks.rados.rados.0.vpm167.stdout:3049: seq_num 1150 ranges {502233=435442,1455477=485407,2591408=310825} 2014-08-08T10:40:10.224 INFO:tasks.rados.rados.0.vpm167.stdout:3049: writing vpm1675861-12 from 502233 to 937675 tid 1 2014-08-08T10:40:10.258 INFO:tasks.rados.rados.0.vpm167.stdout:3049: writing vpm1675861-12 from 1455477 to 1940884 tid 2 2014-08-08T10:40:10.284 INFO:tasks.rados.rados.0.vpm167.stdout:3049: writing vpm1675861-12 from 2591408 to 2902233 tid 3 2014-08-08T10:40:10.290 INFO:tasks.rados.rados.0.vpm167.stdout:3049: finishing write tid 1 to vpm1675861-12 2014-08-08T10:40:10.290 INFO:tasks.rados.rados.0.vpm167.stdout:3049: finishing write tid 2 to vpm1675861-12 2014-08-08T10:40:10.290 INFO:tasks.rados.rados.0.vpm167.stdout:3050: write oid 38 current snap is 406 2014-08-08T10:40:10.290 INFO:tasks.rados.rados.0.vpm167.stdout:3050: seq_num 1151 ranges {597463=655822,1817161=707026,2597462=1} 2014-08-08T10:40:10.328 INFO:tasks.rados.rados.0.vpm167.stdout:3050: writing vpm1675861-38 from 597463 to 1253285 tid 1 2014-08-08T10:40:10.375 INFO:tasks.rados.rados.0.vpm167.stdout:3050: writing vpm1675861-38 from 1817161 to 2524187 tid 2 2014-08-08T10:40:10.377 INFO:tasks.rados.rados.0.vpm167.stdout:3050: writing vpm1675861-38 from 2597462 to 2597463 tid 3 2014-08-08T10:40:10.378 INFO:tasks.rados.rados.0.vpm167.stdout:3049: finishing write tid 3 to vpm1675861-12 2014-08-08T10:40:10.378 INFO:tasks.rados.rados.0.vpm167.stdout:3049: finishing write tid 5 to vpm1675861-12 2014-08-08T10:40:10.378 INFO:tasks.rados.rados.0.vpm167.stdout:3049: finishing write tid 6 to vpm1675861-12 2014-08-08T10:40:10.379 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2067 (ObjNum 1150 snap 406 seq_num 1150) dirty exists 2014-08-08T10:40:10.379 INFO:tasks.rados.rados.0.vpm167.stdout:3050: finishing write tid 1 to vpm1675861-38 2014-08-08T10:40:10.379 INFO:tasks.rados.rados.0.vpm167.stdout:3049: done (1 left) 2014-08-08T10:40:10.379 INFO:tasks.rados.rados.0.vpm167.stdout:3051: delete oid 44 current snap is 406 2014-08-08T10:40:10.389 INFO:tasks.rados.rados.0.vpm167.stdout:3050: finishing write tid 2 to vpm1675861-38 2014-08-08T10:40:10.389 INFO:tasks.rados.rados.0.vpm167.stdout:3050: finishing write tid 3 to vpm1675861-38 2014-08-08T10:40:10.389 INFO:tasks.rados.rados.0.vpm167.stdout:3050: finishing write tid 5 to vpm1675861-38 2014-08-08T10:40:10.391 INFO:tasks.rados.rados.0.vpm167.stdout:3050: finishing write tid 6 to vpm1675861-38 2014-08-08T10:40:10.391 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 38 v 798 (ObjNum 1151 snap 406 seq_num 1151) dirty exists 2014-08-08T10:40:10.451 INFO:tasks.rados.rados.0.vpm167.stdout:3050: done (1 left) 2014-08-08T10:40:10.451 INFO:tasks.rados.rados.0.vpm167.stdout:3051: done (0 left) 2014-08-08T10:40:10.451 INFO:tasks.rados.rados.0.vpm167.stdout:3052: rollback oid 23 current snap is 406 2014-08-08T10:40:10.451 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 23 to 370 2014-08-08T10:40:10.452 INFO:tasks.rados.rados.0.vpm167.stdout:3053: snap_create 2014-08-08T10:40:10.470 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 23 v 823 (ObjNum 1069 snap 370 seq_num 1069) dirty exists 2014-08-08T10:40:11.241 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.2. 2014-08-08T10:40:11.326 INFO:tasks.rados.rados.0.vpm167.stdout:3052: done (1 left) 2014-08-08T10:40:11.326 INFO:tasks.rados.rados.0.vpm167.stdout:3053: done (0 left) 2014-08-08T10:40:11.326 INFO:tasks.rados.rados.0.vpm167.stdout:3054: read oid 46 snap -1 2014-08-08T10:40:11.326 INFO:tasks.rados.rados.0.vpm167.stdout:3055: read oid 12 snap 397 2014-08-08T10:40:11.327 INFO:tasks.rados.rados.0.vpm167.stdout:3056: write oid 29 current snap is 407 2014-08-08T10:40:11.327 INFO:tasks.rados.rados.0.vpm167.stdout:3056: seq_num 1153 ranges {86098=1} 2014-08-08T10:40:11.328 INFO:tasks.rados.rados.0.vpm167.stdout:3056: writing vpm1675861-29 from 86098 to 86099 tid 1 2014-08-08T10:40:11.334 INFO:tasks.rados.rados.0.vpm167.stdout:3057: snap_remove snap 398 2014-08-08T10:40:11.339 INFO:tasks.rados.rados.0.vpm167.stdout:3055: expect (ObjNum 950 snap 312 seq_num 950) 2014-08-08T10:40:12.169 INFO:tasks.rados.rados.0.vpm167.stdout:3056: finishing write tid 1 to vpm1675861-29 2014-08-08T10:40:12.169 INFO:tasks.rados.rados.0.vpm167.stdout:3056: finishing write tid 3 to vpm1675861-29 2014-08-08T10:40:12.170 INFO:tasks.rados.rados.0.vpm167.stdout:3054: expect (ObjNum 1060 snap 364 seq_num 1060) 2014-08-08T10:40:13.095 INFO:tasks.rados.rados.0.vpm167.stdout:3054: done (3 left) 2014-08-08T10:40:14.742 INFO:tasks.rados.rados.0.vpm167.stdout:3055: done (2 left) 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:3057: done (1 left) 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:3058: delete oid 32 current snap is 407 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:3056: finishing write tid 4 to vpm1675861-29 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 949 (ObjNum 1153 snap 407 seq_num 1153) dirty exists 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:3056: done (1 left) 2014-08-08T10:40:14.744 INFO:tasks.rados.rados.0.vpm167.stdout:3058: done (0 left) 2014-08-08T10:40:14.745 INFO:tasks.rados.rados.0.vpm167.stdout:3059: read oid 4 snap 344 2014-08-08T10:40:14.745 INFO:tasks.rados.rados.0.vpm167.stdout:3060: read oid 38 snap -1 2014-08-08T10:40:14.745 INFO:tasks.rados.rados.0.vpm167.stdout:3061: snap_remove snap 375 2014-08-08T10:40:14.745 INFO:tasks.rados.rados.0.vpm167.stdout:3059: expect (ObjNum 1008 snap 343 seq_num 1008) 2014-08-08T10:40:14.745 INFO:tasks.rados.rados.0.vpm167.stdout:3060: expect (ObjNum 1151 snap 406 seq_num 1151) 2014-08-08T10:40:14.836 INFO:tasks.rados.rados.0.vpm167.stdout:3059: done (2 left) 2014-08-08T10:40:14.837 INFO:tasks.rados.rados.0.vpm167.stdout:3060: done (1 left) 2014-08-08T10:40:14.837 INFO:tasks.rados.rados.0.vpm167.stdout:3061: done (0 left) 2014-08-08T10:40:14.838 INFO:tasks.rados.rados.0.vpm167.stdout:3062: delete oid 18 current snap is 407 2014-08-08T10:40:16.267 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5] out_osds: [2] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:16.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:16.268 INFO:tasks.thrashosds.thrasher:fixing pg num pool data 2014-08-08T10:40:16.268 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:40:16.438 INFO:tasks.rados.rados.0.vpm167.stdout:3062: done (0 left) 2014-08-08T10:40:16.438 INFO:tasks.rados.rados.0.vpm167.stdout:3063: write oid 7 current snap is 407 2014-08-08T10:40:16.438 INFO:tasks.rados.rados.0.vpm167.stdout:3063: seq_num 1156 ranges {587340=453788,1764206=23134} 2014-08-08T10:40:16.452 INFO:tasks.rados.rados.0.vpm167.stdout:3063: writing vpm1675861-7 from 587340 to 1041128 tid 1 2014-08-08T10:40:16.513 INFO:tasks.rados.rados.0.vpm167.stdout:3063: writing vpm1675861-7 from 1764206 to 1787340 tid 2 2014-08-08T10:40:16.513 INFO:tasks.rados.rados.0.vpm167.stdout:3064: write oid 38 current snap is 407 2014-08-08T10:40:16.514 INFO:tasks.rados.rados.0.vpm167.stdout:3064: seq_num 1157 ranges {682570=426616,1482569=1} 2014-08-08T10:40:16.534 INFO:tasks.rados.rados.0.vpm167.stdout:3064: writing vpm1675861-38 from 682570 to 1109186 tid 1 2014-08-08T10:40:16.541 INFO:tasks.rados.rados.0.vpm167.stdout:3064: writing vpm1675861-38 from 1482569 to 1482570 tid 2 2014-08-08T10:40:16.542 INFO:tasks.rados.rados.0.vpm167.stdout:3065: read oid 24 snap 370 2014-08-08T10:40:16.542 INFO:tasks.rados.rados.0.vpm167.stdout:3066: snap_remove snap 402 2014-08-08T10:40:16.586 INFO:tasks.rados.rados.0.vpm167.stdout:3064: finishing write tid 1 to vpm1675861-38 2014-08-08T10:40:16.586 INFO:tasks.rados.rados.0.vpm167.stdout:3064: finishing write tid 2 to vpm1675861-38 2014-08-08T10:40:16.587 INFO:tasks.rados.rados.0.vpm167.stdout:3064: finishing write tid 4 to vpm1675861-38 2014-08-08T10:40:16.592 INFO:tasks.rados.rados.0.vpm167.stdout:3063: finishing write tid 1 to vpm1675861-7 2014-08-08T10:40:16.605 INFO:tasks.rados.rados.0.vpm167.stdout:3063: finishing write tid 2 to vpm1675861-7 2014-08-08T10:40:16.610 INFO:tasks.rados.rados.0.vpm167.stdout:3064: finishing write tid 5 to vpm1675861-38 2014-08-08T10:40:16.610 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 38 v 802 (ObjNum 1157 snap 407 seq_num 1157) dirty exists 2014-08-08T10:40:16.621 INFO:tasks.rados.rados.0.vpm167.stdout:3063: finishing write tid 4 to vpm1675861-7 2014-08-08T10:40:16.622 INFO:tasks.rados.rados.0.vpm167.stdout:3063: finishing write tid 5 to vpm1675861-7 2014-08-08T10:40:16.622 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 726 (ObjNum 1156 snap 407 seq_num 1156) dirty exists 2014-08-08T10:40:16.903 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:40:16.957 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 44' 2014-08-08T10:40:17.634 INFO:tasks.rados.rados.0.vpm167.stdout:3063: done (3 left) 2014-08-08T10:40:17.634 INFO:tasks.rados.rados.0.vpm167.stdout:3064: done (2 left) 2014-08-08T10:40:17.635 INFO:tasks.rados.rados.0.vpm167.stdout:3065: done (1 left) 2014-08-08T10:40:17.635 INFO:tasks.rados.rados.0.vpm167.stdout:3066: done (0 left) 2014-08-08T10:40:17.635 INFO:tasks.rados.rados.0.vpm167.stdout:3067: read oid 39 snap -1 2014-08-08T10:40:17.635 INFO:tasks.rados.rados.0.vpm167.stdout:3068: delete oid 1 current snap is 407 2014-08-08T10:40:17.646 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 0 pgp_num to 44 2014-08-08T10:40:17.835 INFO:tasks.rados.rados.0.vpm167.stdout:3068: done (1 left) 2014-08-08T10:40:17.835 INFO:tasks.rados.rados.0.vpm167.stdout:3069: write oid 35 current snap is 407 2014-08-08T10:40:17.836 INFO:tasks.rados.rados.0.vpm167.stdout:3069: seq_num 1159 ranges {687557=405297,1518058=654059,2725892=714692,3487556=1} 2014-08-08T10:40:17.863 INFO:tasks.rados.rados.0.vpm167.stdout:3069: writing vpm1675861-35 from 687557 to 1092854 tid 1 2014-08-08T10:40:17.903 INFO:tasks.rados.rados.0.vpm167.stdout:3069: writing vpm1675861-35 from 1518058 to 2172117 tid 2 2014-08-08T10:40:17.947 INFO:tasks.rados.rados.0.vpm167.stdout:3069: writing vpm1675861-35 from 2725892 to 3440584 tid 3 2014-08-08T10:40:17.950 INFO:tasks.rados.rados.0.vpm167.stdout:3069: writing vpm1675861-35 from 3487556 to 3487557 tid 4 2014-08-08T10:40:17.951 INFO:tasks.rados.rados.0.vpm167.stdout:3067: done (1 left) 2014-08-08T10:40:17.952 INFO:tasks.rados.rados.0.vpm167.stdout:3070: snap_create 2014-08-08T10:40:18.232 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 1 to vpm1675861-35 2014-08-08T10:40:18.233 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 2 to vpm1675861-35 2014-08-08T10:40:18.588 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 3 to vpm1675861-35 2014-08-08T10:40:18.589 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 4 to vpm1675861-35 2014-08-08T10:40:18.589 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 6 to vpm1675861-35 2014-08-08T10:40:18.600 INFO:tasks.rados.rados.0.vpm167.stdout:3069: finishing write tid 7 to vpm1675861-35 2014-08-08T10:40:18.600 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 35 v 969 (ObjNum 1159 snap 407 seq_num 1159) dirty exists 2014-08-08T10:40:18.960 INFO:tasks.rados.rados.0.vpm167.stdout:3069: done (1 left) 2014-08-08T10:40:18.960 INFO:tasks.rados.rados.0.vpm167.stdout:3070: done (0 left) 2014-08-08T10:40:18.960 INFO:tasks.rados.rados.0.vpm167.stdout:3071: rollback oid 6 current snap is 408 2014-08-08T10:40:18.961 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 6 to 386 2014-08-08T10:40:18.961 INFO:tasks.rados.rados.0.vpm167.stdout:3072: snap_remove snap 368 2014-08-08T10:40:19.459 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 6 v 1647 (ObjNum 1006 snap 342 seq_num 1006) dirty exists 2014-08-08T10:40:19.975 INFO:tasks.rados.rados.0.vpm167.stdout:3071: done (1 left) 2014-08-08T10:40:19.975 INFO:tasks.rados.rados.0.vpm167.stdout:3072: done (0 left) 2014-08-08T10:40:19.975 INFO:tasks.rados.rados.0.vpm167.stdout:3073: read oid 19 snap -1 2014-08-08T10:40:19.976 INFO:tasks.rados.rados.0.vpm167.stdout:3074: snap_remove snap 405 2014-08-08T10:40:20.148 INFO:tasks.rados.rados.0.vpm167.stdout:3073: expect (ObjNum 1125 snap 396 seq_num 1125) 2014-08-08T10:40:21.016 INFO:tasks.rados.rados.0.vpm167.stdout:3073: done (1 left) 2014-08-08T10:40:21.017 INFO:tasks.rados.rados.0.vpm167.stdout:3074: done (0 left) 2014-08-08T10:40:21.017 INFO:tasks.rados.rados.0.vpm167.stdout:3075: read oid 19 snap -1 2014-08-08T10:40:21.017 INFO:tasks.rados.rados.0.vpm167.stdout:3076: write oid 3 current snap is 408 2014-08-08T10:40:21.018 INFO:tasks.rados.rados.0.vpm167.stdout:3076: seq_num 1160 ranges {782787=528525,1898942=475679,2795918=386869} 2014-08-08T10:40:21.055 INFO:tasks.rados.rados.0.vpm167.stdout:3076: writing vpm1675861-3 from 782787 to 1311312 tid 1 2014-08-08T10:40:21.093 INFO:tasks.rados.rados.0.vpm167.stdout:3076: writing vpm1675861-3 from 1898942 to 2374621 tid 2 2014-08-08T10:40:21.119 INFO:tasks.rados.rados.0.vpm167.stdout:3076: writing vpm1675861-3 from 2795918 to 3182787 tid 3 2014-08-08T10:40:21.121 INFO:tasks.rados.rados.0.vpm167.stdout:3076: finishing write tid 1 to vpm1675861-3 2014-08-08T10:40:21.121 INFO:tasks.rados.rados.0.vpm167.stdout:3077: read oid 20 snap 376 2014-08-08T10:40:21.122 INFO:tasks.rados.rados.0.vpm167.stdout:3078: read oid 29 snap -1 2014-08-08T10:40:21.122 INFO:tasks.rados.rados.0.vpm167.stdout:3079: rollback oid 33 current snap is 408 2014-08-08T10:40:21.123 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 33 to 399 2014-08-08T10:40:21.123 INFO:tasks.rados.rados.0.vpm167.stdout:3080: read oid 47 snap -1 2014-08-08T10:40:21.124 INFO:tasks.rados.rados.0.vpm167.stdout:3081: write oid 31 current snap is 408 2014-08-08T10:40:21.124 INFO:tasks.rados.rados.0.vpm167.stdout:3081: seq_num 1161 ranges {727616=501354,1862674=545874,2846743=735825,3927615=1} 2014-08-08T10:40:21.160 INFO:tasks.rados.rados.0.vpm167.stdout:3081: writing vpm1675861-31 from 727616 to 1228970 tid 1 2014-08-08T10:40:21.198 INFO:tasks.rados.rados.0.vpm167.stdout:3081: writing vpm1675861-31 from 1862674 to 2408548 tid 2 2014-08-08T10:40:21.254 INFO:tasks.rados.rados.0.vpm167.stdout:3081: writing vpm1675861-31 from 2846743 to 3582568 tid 3 2014-08-08T10:40:21.260 INFO:tasks.rados.rados.0.vpm167.stdout:3081: writing vpm1675861-31 from 3927615 to 3927616 tid 4 2014-08-08T10:40:21.261 INFO:tasks.rados.rados.0.vpm167.stdout:3076: finishing write tid 2 to vpm1675861-3 2014-08-08T10:40:21.261 INFO:tasks.rados.rados.0.vpm167.stdout:3078: expect (ObjNum 1153 snap 407 seq_num 1153) 2014-08-08T10:40:21.272 INFO:tasks.rados.rados.0.vpm167.stdout:3077: done (6 left) 2014-08-08T10:40:21.272 INFO:tasks.rados.rados.0.vpm167.stdout:3078: done (5 left) 2014-08-08T10:40:21.272 INFO:tasks.rados.rados.0.vpm167.stdout:3082: read oid 39 snap -1 2014-08-08T10:40:21.273 INFO:tasks.rados.rados.0.vpm167.stdout:3083: snap_remove snap 403 2014-08-08T10:40:21.273 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 1 to vpm1675861-31 2014-08-08T10:40:21.273 INFO:tasks.rados.rados.0.vpm167.stdout:3076: finishing write tid 3 to vpm1675861-3 2014-08-08T10:40:21.274 INFO:tasks.rados.rados.0.vpm167.stdout:3076: finishing write tid 5 to vpm1675861-3 2014-08-08T10:40:21.274 INFO:tasks.rados.rados.0.vpm167.stdout:3076: finishing write tid 6 to vpm1675861-3 2014-08-08T10:40:21.274 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 739 (ObjNum 1160 snap 408 seq_num 1160) dirty exists 2014-08-08T10:40:21.274 INFO:tasks.rados.rados.0.vpm167.stdout:3075: expect (ObjNum 1125 snap 396 seq_num 1125) 2014-08-08T10:40:21.932 INFO:tasks.rados.rados.0.vpm167.stdout:3080: expect (ObjNum 1124 snap 396 seq_num 1124) 2014-08-08T10:40:22.507 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1666 (ObjNum 1129 snap 398 seq_num 1129) dirty exists 2014-08-08T10:40:22.507 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 2 to vpm1675861-31 2014-08-08T10:40:22.508 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 3 to vpm1675861-31 2014-08-08T10:40:22.508 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 4 to vpm1675861-31 2014-08-08T10:40:22.508 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 6 to vpm1675861-31 2014-08-08T10:40:22.577 INFO:tasks.rados.rados.0.vpm167.stdout:3081: finishing write tid 7 to vpm1675861-31 2014-08-08T10:40:22.577 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1224 (ObjNum 1161 snap 408 seq_num 1161) dirty exists 2014-08-08T10:40:22.672 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5] out_osds: [2] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:22.673 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:22.673 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2014-08-08T10:40:22.673 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:40:22.674 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:40:22.674 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-08-08T10:40:22.766 INFO:tasks.rados.rados.0.vpm167.stdout:3075: done (6 left) 2014-08-08T10:40:22.766 INFO:tasks.rados.rados.0.vpm167.stdout:3076: done (5 left) 2014-08-08T10:40:22.766 INFO:tasks.rados.rados.0.vpm167.stdout:3079: done (4 left) 2014-08-08T10:40:22.767 INFO:tasks.rados.rados.0.vpm167.stdout:3080: done (3 left) 2014-08-08T10:40:22.767 INFO:tasks.rados.rados.0.vpm167.stdout:3081: done (2 left) 2014-08-08T10:40:22.767 INFO:tasks.rados.rados.0.vpm167.stdout:3082: done (1 left) 2014-08-08T10:40:22.767 INFO:tasks.rados.rados.0.vpm167.stdout:3083: done (0 left) 2014-08-08T10:40:22.768 INFO:tasks.rados.rados.0.vpm167.stdout:3084: snap_remove snap 392 2014-08-08T10:40:24.197 INFO:tasks.rados.rados.0.vpm167.stdout:3084: done (0 left) 2014-08-08T10:40:24.197 INFO:tasks.rados.rados.0.vpm167.stdout:3085: snap_remove snap 404 2014-08-08T10:40:25.487 INFO:tasks.rados.rados.0.vpm167.stdout:3085: done (0 left) 2014-08-08T10:40:25.488 INFO:tasks.rados.rados.0.vpm167.stdout:3086: snap_create 2014-08-08T10:40:26.513 INFO:tasks.rados.rados.0.vpm167.stdout:3086: done (0 left) 2014-08-08T10:40:26.513 INFO:tasks.rados.rados.0.vpm167.stdout:3087: snap_remove snap 382 2014-08-08T10:40:27.598 INFO:tasks.rados.rados.0.vpm167.stdout:3087: done (0 left) 2014-08-08T10:40:27.598 INFO:tasks.rados.rados.0.vpm167.stdout:3088: write oid 44 current snap is 409 2014-08-08T10:40:27.598 INFO:tasks.rados.rados.0.vpm167.stdout:3088: seq_num 1162 ranges {671421=1} 2014-08-08T10:40:27.604 INFO:tasks.rados.rados.0.vpm167.stdout:3088: writing vpm1675861-44 from 671421 to 671422 tid 1 2014-08-08T10:40:27.605 INFO:tasks.rados.rados.0.vpm167.stdout:3089: rollback oid 3 current snap is 409 2014-08-08T10:40:27.605 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 389 2014-08-08T10:40:27.605 INFO:tasks.rados.rados.0.vpm167.stdout:3090: read oid 6 snap -1 2014-08-08T10:40:27.606 INFO:tasks.rados.rados.0.vpm167.stdout:3091: delete oid 17 current snap is 409 2014-08-08T10:40:27.631 INFO:tasks.rados.rados.0.vpm167.stdout:3091: done (3 left) 2014-08-08T10:40:27.632 INFO:tasks.rados.rados.0.vpm167.stdout:3092: read oid 34 snap -1 2014-08-08T10:40:27.632 INFO:tasks.rados.rados.0.vpm167.stdout:3093: read oid 32 snap -1 2014-08-08T10:40:27.633 INFO:tasks.rados.rados.0.vpm167.stdout:3094: write oid 8 current snap is 409 2014-08-08T10:40:27.633 INFO:tasks.rados.rados.0.vpm167.stdout:3094: seq_num 1164 ranges {428858=453887,1619600=9258} 2014-08-08T10:40:27.658 INFO:tasks.rados.rados.0.vpm167.stdout:3094: writing vpm1675861-8 from 428858 to 882745 tid 1 2014-08-08T10:40:27.670 INFO:tasks.rados.rados.0.vpm167.stdout:3094: writing vpm1675861-8 from 1619600 to 1628858 tid 2 2014-08-08T10:40:27.671 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 754 (ObjNum 30010208 snap 0 seq_num 30009632) dirty dne 2014-08-08T10:40:27.671 INFO:tasks.rados.rados.0.vpm167.stdout:3089: done (5 left) 2014-08-08T10:40:27.672 INFO:tasks.rados.rados.0.vpm167.stdout:3092: done (4 left) 2014-08-08T10:40:27.672 INFO:tasks.rados.rados.0.vpm167.stdout:3093: done (3 left) 2014-08-08T10:40:27.672 INFO:tasks.rados.rados.0.vpm167.stdout:3095: read oid 9 snap 358 2014-08-08T10:40:27.673 INFO:tasks.rados.rados.0.vpm167.stdout:3096: write oid 12 current snap is 409 2014-08-08T10:40:27.673 INFO:tasks.rados.rados.0.vpm167.stdout:3096: seq_num 1165 ranges {772663=425691,1946211=426452} 2014-08-08T10:40:27.708 INFO:tasks.rados.rados.0.vpm167.stdout:3096: writing vpm1675861-12 from 772663 to 1198354 tid 1 2014-08-08T10:40:27.737 INFO:tasks.rados.rados.0.vpm167.stdout:3096: writing vpm1675861-12 from 1946211 to 2372663 tid 2 2014-08-08T10:40:27.739 INFO:tasks.rados.rados.0.vpm167.stdout:3094: finishing write tid 1 to vpm1675861-8 2014-08-08T10:40:27.740 INFO:tasks.rados.rados.0.vpm167.stdout:3094: finishing write tid 2 to vpm1675861-8 2014-08-08T10:40:27.740 INFO:tasks.rados.rados.0.vpm167.stdout:3095: expect (ObjNum 1033 snap 354 seq_num 1033) 2014-08-08T10:40:28.053 INFO:tasks.rados.rados.0.vpm167.stdout:3095: done (4 left) 2014-08-08T10:40:28.053 INFO:tasks.rados.rados.0.vpm167.stdout:3097: delete oid 40 current snap is 409 2014-08-08T10:40:28.054 INFO:tasks.rados.rados.0.vpm167.stdout:3094: finishing write tid 4 to vpm1675861-8 2014-08-08T10:40:28.054 INFO:tasks.rados.rados.0.vpm167.stdout:3094: finishing write tid 5 to vpm1675861-8 2014-08-08T10:40:28.054 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 8 v 2074 (ObjNum 1164 snap 409 seq_num 1164) dirty exists 2014-08-08T10:40:28.055 INFO:tasks.rados.rados.0.vpm167.stdout:3096: finishing write tid 1 to vpm1675861-12 2014-08-08T10:40:28.055 INFO:tasks.rados.rados.0.vpm167.stdout:3096: finishing write tid 2 to vpm1675861-12 2014-08-08T10:40:28.055 INFO:tasks.rados.rados.0.vpm167.stdout:3096: finishing write tid 4 to vpm1675861-12 2014-08-08T10:40:28.055 INFO:tasks.rados.rados.0.vpm167.stdout:3096: finishing write tid 5 to vpm1675861-12 2014-08-08T10:40:28.056 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2078 (ObjNum 1165 snap 409 seq_num 1165) dirty exists 2014-08-08T10:40:28.056 INFO:tasks.rados.rados.0.vpm167.stdout:3094: done (4 left) 2014-08-08T10:40:28.056 INFO:tasks.rados.rados.0.vpm167.stdout:3096: done (3 left) 2014-08-08T10:40:28.056 INFO:tasks.rados.rados.0.vpm167.stdout:3097: done (2 left) 2014-08-08T10:40:28.057 INFO:tasks.rados.rados.0.vpm167.stdout:3098: read oid 31 snap -1 2014-08-08T10:40:28.057 INFO:tasks.rados.rados.0.vpm167.stdout:3099: snap_remove snap 372 2014-08-08T10:40:28.057 INFO:tasks.rados.rados.0.vpm167.stdout:3088: finishing write tid 1 to vpm1675861-44 2014-08-08T10:40:28.058 INFO:tasks.rados.rados.0.vpm167.stdout:3088: finishing write tid 3 to vpm1675861-44 2014-08-08T10:40:28.058 INFO:tasks.rados.rados.0.vpm167.stdout:3090: expect (ObjNum 1006 snap 342 seq_num 1006) 2014-08-08T10:40:28.131 INFO:tasks.rados.rados.0.vpm167.stdout:3088: finishing write tid 4 to vpm1675861-44 2014-08-08T10:40:28.132 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1384 (ObjNum 1162 snap 409 seq_num 1162) dirty exists 2014-08-08T10:40:28.132 INFO:tasks.rados.rados.0.vpm167.stdout:3098: expect (ObjNum 1161 snap 408 seq_num 1161) 2014-08-08T10:40:29.062 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5] out_osds: [2] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:29.062 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:29.063 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2014-08-08T10:40:29.063 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:40:29.284 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:40:29.342 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T10:40:29.342 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_1 pg_num 46' 2014-08-08T10:40:30.594 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 4 pg_num to 46 2014-08-08T10:40:30.768 INFO:tasks.rados.rados.0.vpm167.stdout:3088: done (3 left) 2014-08-08T10:40:30.768 INFO:tasks.rados.rados.0.vpm167.stdout:3090: done (2 left) 2014-08-08T10:40:30.769 INFO:tasks.rados.rados.0.vpm167.stdout:3098: done (1 left) 2014-08-08T10:40:30.769 INFO:tasks.rados.rados.0.vpm167.stdout:3099: done (0 left) 2014-08-08T10:40:30.769 INFO:tasks.rados.rados.0.vpm167.stdout:3100: delete oid 18 current snap is 409 2014-08-08T10:40:30.771 INFO:tasks.rados.rados.0.vpm167.stdout:3100: done (0 left) 2014-08-08T10:40:30.771 INFO:tasks.rados.rados.0.vpm167.stdout:3101: rollback oid 28 current snap is 409 2014-08-08T10:40:30.772 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 28 to 396 2014-08-08T10:40:30.772 INFO:tasks.rados.rados.0.vpm167.stdout:3102: read oid 26 snap 388 2014-08-08T10:40:30.772 INFO:tasks.rados.rados.0.vpm167.stdout:3103: snap_create 2014-08-08T10:40:30.781 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 0 (ObjNum 44425648 snap 0 seq_num 48934096) dirty dne 2014-08-08T10:40:30.874 INFO:tasks.rados.rados.0.vpm167.stdout:3102: expect (ObjNum 1071 snap 371 seq_num 1071) 2014-08-08T10:40:31.751 INFO:tasks.rados.rados.0.vpm167.stdout:3101: done (2 left) 2014-08-08T10:40:31.752 INFO:tasks.rados.rados.0.vpm167.stdout:3102: done (1 left) 2014-08-08T10:40:31.752 INFO:tasks.rados.rados.0.vpm167.stdout:3103: done (0 left) 2014-08-08T10:40:31.752 INFO:tasks.rados.rados.0.vpm167.stdout:3104: write oid 14 current snap is 410 2014-08-08T10:40:31.752 INFO:tasks.rados.rados.0.vpm167.stdout:3104: seq_num 1168 ranges {72880=1} 2014-08-08T10:40:31.753 INFO:tasks.rados.rados.0.vpm167.stdout:3104: writing vpm1675861-14 from 72880 to 72881 tid 1 2014-08-08T10:40:31.753 INFO:tasks.rados.rados.0.vpm167.stdout:3105: write oid 40 current snap is 410 2014-08-08T10:40:31.754 INFO:tasks.rados.rados.0.vpm167.stdout:3105: seq_num 1169 ranges {569134=500428,1565619=645355,2935522=665862,3769133=1} 2014-08-08T10:40:31.781 INFO:tasks.rados.rados.0.vpm167.stdout:3105: writing vpm1675861-40 from 569134 to 1069562 tid 1 2014-08-08T10:40:31.829 INFO:tasks.rados.rados.0.vpm167.stdout:3105: writing vpm1675861-40 from 1565619 to 2210974 tid 2 2014-08-08T10:40:31.863 INFO:tasks.rados.rados.0.vpm167.stdout:3105: writing vpm1675861-40 from 2935522 to 3601384 tid 3 2014-08-08T10:40:31.867 INFO:tasks.rados.rados.0.vpm167.stdout:3105: writing vpm1675861-40 from 3769133 to 3769134 tid 4 2014-08-08T10:40:31.868 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 1 to vpm1675861-40 2014-08-08T10:40:31.868 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 2 to vpm1675861-40 2014-08-08T10:40:31.869 INFO:tasks.rados.rados.0.vpm167.stdout:3106: write oid 19 current snap is 410 2014-08-08T10:40:31.869 INFO:tasks.rados.rados.0.vpm167.stdout:3106: seq_num 1170 ranges {512939=1} 2014-08-08T10:40:31.874 INFO:tasks.rados.rados.0.vpm167.stdout:3106: writing vpm1675861-19 from 512939 to 512940 tid 1 2014-08-08T10:40:31.875 INFO:tasks.rados.rados.0.vpm167.stdout:3107: snap_create 2014-08-08T10:40:31.894 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 3 to vpm1675861-40 2014-08-08T10:40:31.895 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 4 to vpm1675861-40 2014-08-08T10:40:31.898 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 6 to vpm1675861-40 2014-08-08T10:40:31.900 INFO:tasks.rados.rados.0.vpm167.stdout:3105: finishing write tid 7 to vpm1675861-40 2014-08-08T10:40:31.900 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 40 v 781 (ObjNum 1169 snap 410 seq_num 1169) dirty exists 2014-08-08T10:40:31.904 INFO:tasks.rados.rados.0.vpm167.stdout:3104: finishing write tid 1 to vpm1675861-14 2014-08-08T10:40:31.913 INFO:tasks.rados.rados.0.vpm167.stdout:3104: finishing write tid 3 to vpm1675861-14 2014-08-08T10:40:31.913 INFO:tasks.rados.rados.0.vpm167.stdout:3104: finishing write tid 4 to vpm1675861-14 2014-08-08T10:40:31.913 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1948 (ObjNum 1168 snap 410 seq_num 1168) dirty exists 2014-08-08T10:40:31.992 INFO:tasks.rados.rados.0.vpm167.stdout:3106: finishing write tid 1 to vpm1675861-19 2014-08-08T10:40:31.992 INFO:tasks.rados.rados.0.vpm167.stdout:3106: finishing write tid 3 to vpm1675861-19 2014-08-08T10:40:32.103 INFO:tasks.rados.rados.0.vpm167.stdout:3106: finishing write tid 4 to vpm1675861-19 2014-08-08T10:40:32.104 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 19 v 1301 (ObjNum 1170 snap 410 seq_num 1170) dirty exists 2014-08-08T10:40:32.829 INFO:tasks.rados.rados.0.vpm167.stdout:3104: done (3 left) 2014-08-08T10:40:32.830 INFO:tasks.rados.rados.0.vpm167.stdout:3105: done (2 left) 2014-08-08T10:40:32.830 INFO:tasks.rados.rados.0.vpm167.stdout:3106: done (1 left) 2014-08-08T10:40:32.830 INFO:tasks.rados.rados.0.vpm167.stdout:3107: done (0 left) 2014-08-08T10:40:32.830 INFO:tasks.rados.rados.0.vpm167.stdout:3108: snap_create 2014-08-08T10:40:33.860 INFO:tasks.rados.rados.0.vpm167.stdout:3108: done (0 left) 2014-08-08T10:40:33.860 INFO:tasks.rados.rados.0.vpm167.stdout:3109: delete oid 9 current snap is 412 2014-08-08T10:40:33.972 INFO:tasks.rados.rados.0.vpm167.stdout:3109: done (0 left) 2014-08-08T10:40:33.972 INFO:tasks.rados.rados.0.vpm167.stdout:3110: write oid 14 current snap is 412 2014-08-08T10:40:33.973 INFO:tasks.rados.rados.0.vpm167.stdout:3110: seq_num 1172 ranges {553000=400000} 2014-08-08T10:40:33.989 INFO:tasks.rados.rados.0.vpm167.stdout:3110: writing vpm1675861-14 from 553000 to 953000 tid 1 2014-08-08T10:40:33.994 INFO:tasks.rados.rados.0.vpm167.stdout:3111: snap_create 2014-08-08T10:40:34.092 INFO:tasks.rados.rados.0.vpm167.stdout:3110: finishing write tid 1 to vpm1675861-14 2014-08-08T10:40:34.093 INFO:tasks.rados.rados.0.vpm167.stdout:3110: finishing write tid 3 to vpm1675861-14 2014-08-08T10:40:34.094 INFO:tasks.rados.rados.0.vpm167.stdout:3110: finishing write tid 4 to vpm1675861-14 2014-08-08T10:40:34.094 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1951 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:40:34.926 INFO:tasks.rados.rados.0.vpm167.stdout:3110: done (1 left) 2014-08-08T10:40:34.926 INFO:tasks.rados.rados.0.vpm167.stdout:3111: done (0 left) 2014-08-08T10:40:34.927 INFO:tasks.rados.rados.0.vpm167.stdout:3112: snap_remove snap 390 2014-08-08T10:40:35.641 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5] out_osds: [2] dead_osds: [2, 3] live_osds: [1, 0, 5, 4] 2014-08-08T10:40:35.641 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:35.642 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2014-08-08T10:40:35.642 INFO:tasks.ceph.osd.2:Restarting daemon 2014-08-08T10:40:35.642 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-08-08T10:40:35.645 INFO:tasks.ceph.osd.2:Started 2014-08-08T10:40:35.645 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-08-08T10:40:35.749 INFO:teuthology.orchestra.run.vpm166.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2014-08-08T10:40:35.756 INFO:tasks.thrashosds.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2014-08-08T10:40:36.326 INFO:tasks.rados.rados.0.vpm167.stdout:3112: done (0 left) 2014-08-08T10:40:36.326 INFO:tasks.rados.rados.0.vpm167.stdout:3113: read oid 7 snap -1 2014-08-08T10:40:36.326 INFO:tasks.rados.rados.0.vpm167.stdout:3114: write oid 31 current snap is 413 2014-08-08T10:40:36.326 INFO:tasks.rados.rados.0.vpm167.stdout:3114: seq_num 1173 ranges {614181=423742,1799556=414625} 2014-08-08T10:40:36.354 INFO:tasks.rados.rados.0.vpm167.stdout:3114: writing vpm1675861-31 from 614181 to 1037923 tid 1 2014-08-08T10:40:36.374 INFO:tasks.rados.rados.0.vpm167.stdout:3114: writing vpm1675861-31 from 1799556 to 2214181 tid 2 2014-08-08T10:40:36.376 INFO:tasks.rados.rados.0.vpm167.stdout:3113: expect (ObjNum 1156 snap 407 seq_num 1156) 2014-08-08T10:40:36.426 INFO:tasks.ceph.osd.2.vpm166.stdout:starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-08-08T10:40:36.777 INFO:tasks.ceph.osd.2.vpm166.stderr:2014-08-08 17:40:36.776023 7f0d72c608c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-08-08T10:40:36.849 INFO:tasks.rados.rados.0.vpm167.stdout:3113: done (1 left) 2014-08-08T10:40:36.850 INFO:tasks.rados.rados.0.vpm167.stdout:3115: read oid 50 snap -1 2014-08-08T10:40:36.850 INFO:tasks.rados.rados.0.vpm167.stdout:3116: snap_remove snap 400 2014-08-08T10:40:36.851 INFO:tasks.rados.rados.0.vpm167.stdout:3114: finishing write tid 1 to vpm1675861-31 2014-08-08T10:40:36.851 INFO:tasks.rados.rados.0.vpm167.stdout:3114: finishing write tid 2 to vpm1675861-31 2014-08-08T10:40:36.851 INFO:tasks.rados.rados.0.vpm167.stdout:3114: finishing write tid 4 to vpm1675861-31 2014-08-08T10:40:36.851 INFO:tasks.rados.rados.0.vpm167.stdout:3114: finishing write tid 5 to vpm1675861-31 2014-08-08T10:40:36.851 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1228 (ObjNum 1173 snap 413 seq_num 1173) dirty exists 2014-08-08T10:40:36.871 INFO:tasks.rados.rados.0.vpm167.stdout:3115: expect (ObjNum 871 snap 293 seq_num 871) 2014-08-08T10:40:38.957 INFO:tasks.rados.rados.0.vpm167.stdout:3114: done (2 left) 2014-08-08T10:40:38.957 INFO:tasks.rados.rados.0.vpm167.stdout:3115: done (1 left) 2014-08-08T10:40:38.958 INFO:tasks.rados.rados.0.vpm167.stdout:3116: done (0 left) 2014-08-08T10:40:38.958 INFO:tasks.rados.rados.0.vpm167.stdout:3117: write oid 25 current snap is 413 2014-08-08T10:40:38.958 INFO:tasks.rados.rados.0.vpm167.stdout:3117: seq_num 1174 ranges {710435=546970,1910434=1} 2014-08-08T10:40:38.993 INFO:tasks.rados.rados.0.vpm167.stdout:3117: writing vpm1675861-25 from 710435 to 1257405 tid 1 2014-08-08T10:40:39.003 INFO:tasks.rados.rados.0.vpm167.stdout:3117: writing vpm1675861-25 from 1910434 to 1910435 tid 2 2014-08-08T10:40:39.004 INFO:tasks.rados.rados.0.vpm167.stdout:3118: snap_remove snap 385 2014-08-08T10:40:39.024 INFO:tasks.rados.rados.0.vpm167.stdout:3117: finishing write tid 1 to vpm1675861-25 2014-08-08T10:40:39.025 INFO:tasks.rados.rados.0.vpm167.stdout:3117: finishing write tid 2 to vpm1675861-25 2014-08-08T10:40:39.025 INFO:tasks.rados.rados.0.vpm167.stdout:3117: finishing write tid 4 to vpm1675861-25 2014-08-08T10:40:39.034 INFO:tasks.rados.rados.0.vpm167.stdout:3117: finishing write tid 5 to vpm1675861-25 2014-08-08T10:40:39.034 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 25 v 787 (ObjNum 1174 snap 413 seq_num 1174) dirty exists 2014-08-08T10:40:39.844 INFO:tasks.rados.rados.0.vpm167.stdout:3117: done (1 left) 2014-08-08T10:40:39.844 INFO:tasks.rados.rados.0.vpm167.stdout:3118: done (0 left) 2014-08-08T10:40:39.845 INFO:tasks.rados.rados.0.vpm167.stdout:3119: write oid 24 current snap is 413 2014-08-08T10:40:39.845 INFO:tasks.rados.rados.0.vpm167.stdout:3119: seq_num 1175 ranges {653217=403446,1474348=451998,2520602=132615} 2014-08-08T10:40:39.863 INFO:tasks.rados.rados.0.vpm167.stdout:3119: writing vpm1675861-24 from 653217 to 1056663 tid 1 2014-08-08T10:40:39.883 INFO:tasks.rados.rados.0.vpm167.stdout:3119: writing vpm1675861-24 from 1474348 to 1926346 tid 2 2014-08-08T10:40:39.895 INFO:tasks.rados.rados.0.vpm167.stdout:3119: writing vpm1675861-24 from 2520602 to 2653217 tid 3 2014-08-08T10:40:39.896 INFO:tasks.rados.rados.0.vpm167.stdout:3119: finishing write tid 1 to vpm1675861-24 2014-08-08T10:40:39.897 INFO:tasks.rados.rados.0.vpm167.stdout:3119: finishing write tid 2 to vpm1675861-24 2014-08-08T10:40:39.897 INFO:tasks.rados.rados.0.vpm167.stdout:3120: snap_create 2014-08-08T10:40:39.906 INFO:tasks.rados.rados.0.vpm167.stdout:3119: finishing write tid 3 to vpm1675861-24 2014-08-08T10:40:39.907 INFO:tasks.rados.rados.0.vpm167.stdout:3119: finishing write tid 5 to vpm1675861-24 2014-08-08T10:40:39.910 INFO:tasks.rados.rados.0.vpm167.stdout:3119: finishing write tid 6 to vpm1675861-24 2014-08-08T10:40:39.910 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 24 v 1132 (ObjNum 1175 snap 413 seq_num 1175) dirty exists 2014-08-08T10:40:40.755 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-08-08T10:40:41.025 INFO:tasks.rados.rados.0.vpm167.stdout:3119: done (1 left) 2014-08-08T10:40:41.025 INFO:tasks.rados.rados.0.vpm167.stdout:3120: done (0 left) 2014-08-08T10:40:41.025 INFO:tasks.rados.rados.0.vpm167.stdout:3121: delete oid 11 current snap is 414 2014-08-08T10:40:41.095 INFO:tasks.rados.rados.0.vpm167.stdout:3121: done (0 left) 2014-08-08T10:40:41.096 INFO:tasks.rados.rados.0.vpm167.stdout:3122: rollback oid 10 current snap is 414 2014-08-08T10:40:41.096 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 10 to 389 2014-08-08T10:40:41.097 INFO:tasks.rados.rados.0.vpm167.stdout:3123: snap_create 2014-08-08T10:40:41.099 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 10 v 0 (ObjNum 44772688 snap 0 seq_num 31115376) dirty dne 2014-08-08T10:40:42.128 INFO:tasks.rados.rados.0.vpm167.stdout:3122: done (1 left) 2014-08-08T10:40:42.129 INFO:tasks.rados.rados.0.vpm167.stdout:3123: done (0 left) 2014-08-08T10:40:42.129 INFO:tasks.rados.rados.0.vpm167.stdout:3124: write oid 35 current snap is 415 2014-08-08T10:40:42.129 INFO:tasks.rados.rados.0.vpm167.stdout:3124: seq_num 1177 ranges {410652=498479,1667540=592389,3022253=480570,3610651=1} 2014-08-08T10:40:42.148 INFO:tasks.rados.rados.0.vpm167.stdout:3124: writing vpm1675861-35 from 410652 to 909131 tid 1 2014-08-08T10:40:42.177 INFO:tasks.rados.rados.0.vpm167.stdout:3124: writing vpm1675861-35 from 1667540 to 2259929 tid 2 2014-08-08T10:40:42.206 INFO:tasks.rados.rados.0.vpm167.stdout:3124: writing vpm1675861-35 from 3022253 to 3502823 tid 3 2014-08-08T10:40:42.208 INFO:tasks.rados.rados.0.vpm167.stdout:3124: writing vpm1675861-35 from 3610651 to 3610652 tid 4 2014-08-08T10:40:42.208 INFO:tasks.rados.rados.0.vpm167.stdout:3125: rollback oid 13 current snap is 415 2014-08-08T10:40:42.209 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 13 to 408 2014-08-08T10:40:42.209 INFO:tasks.rados.rados.0.vpm167.stdout:3126: delete oid 18 current snap is 415 2014-08-08T10:40:42.212 INFO:tasks.rados.rados.0.vpm167.stdout:3126: done (2 left) 2014-08-08T10:40:42.213 INFO:tasks.rados.rados.0.vpm167.stdout:3127: read oid 37 snap -1 2014-08-08T10:40:42.213 INFO:tasks.rados.rados.0.vpm167.stdout:3128: read oid 16 snap 358 2014-08-08T10:40:42.213 INFO:tasks.rados.rados.0.vpm167.stdout:3129: delete oid 3 current snap is 415 2014-08-08T10:40:42.214 INFO:tasks.rados.rados.0.vpm167.stdout:3127: done (4 left) 2014-08-08T10:40:42.214 INFO:tasks.rados.rados.0.vpm167.stdout:3129: done (3 left) 2014-08-08T10:40:42.214 INFO:tasks.rados.rados.0.vpm167.stdout:3130: snap_remove snap 407 2014-08-08T10:40:42.613 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 1 to vpm1675861-35 2014-08-08T10:40:42.621 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 2 to vpm1675861-35 2014-08-08T10:40:42.624 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1668 (ObjNum 1116 snap 395 seq_num 1116) dirty exists 2014-08-08T10:40:42.680 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 3 to vpm1675861-35 2014-08-08T10:40:42.684 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 4 to vpm1675861-35 2014-08-08T10:40:42.687 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 6 to vpm1675861-35 2014-08-08T10:40:42.796 INFO:tasks.rados.rados.0.vpm167.stdout:3124: finishing write tid 7 to vpm1675861-35 2014-08-08T10:40:42.796 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 35 v 1028 (ObjNum 1177 snap 415 seq_num 1177) dirty exists 2014-08-08T10:40:43.193 INFO:tasks.rados.rados.0.vpm167.stdout:3124: done (3 left) 2014-08-08T10:40:43.193 INFO:tasks.rados.rados.0.vpm167.stdout:3125: done (2 left) 2014-08-08T10:40:43.193 INFO:tasks.rados.rados.0.vpm167.stdout:3128: done (1 left) 2014-08-08T10:40:43.194 INFO:tasks.rados.rados.0.vpm167.stdout:3130: done (0 left) 2014-08-08T10:40:43.194 INFO:tasks.rados.rados.0.vpm167.stdout:3131: write oid 17 current snap is 415 2014-08-08T10:40:43.194 INFO:tasks.rados.rados.0.vpm167.stdout:3131: seq_num 1180 ranges {794516=1} 2014-08-08T10:40:43.201 INFO:tasks.rados.rados.0.vpm167.stdout:3131: writing vpm1675861-17 from 794516 to 794517 tid 1 2014-08-08T10:40:43.202 INFO:tasks.rados.rados.0.vpm167.stdout:3132: read oid 46 snap -1 2014-08-08T10:40:43.203 INFO:tasks.rados.rados.0.vpm167.stdout:3133: write oid 1 current snap is 415 2014-08-08T10:40:43.203 INFO:tasks.rados.rados.0.vpm167.stdout:3133: seq_num 1181 ranges {739347=573216,1539346=1} 2014-08-08T10:40:43.240 INFO:tasks.rados.rados.0.vpm167.stdout:3133: writing vpm1675861-1 from 739347 to 1312563 tid 1 2014-08-08T10:40:43.246 INFO:tasks.rados.rados.0.vpm167.stdout:3133: writing vpm1675861-1 from 1539346 to 1539347 tid 2 2014-08-08T10:40:43.246 INFO:tasks.rados.rados.0.vpm167.stdout:3131: finishing write tid 1 to vpm1675861-17 2014-08-08T10:40:43.246 INFO:tasks.rados.rados.0.vpm167.stdout:3131: finishing write tid 3 to vpm1675861-17 2014-08-08T10:40:43.247 INFO:tasks.rados.rados.0.vpm167.stdout:3131: finishing write tid 4 to vpm1675861-17 2014-08-08T10:40:43.247 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2081 (ObjNum 1180 snap 415 seq_num 1180) dirty exists 2014-08-08T10:40:43.248 INFO:tasks.rados.rados.0.vpm167.stdout:3131: done (2 left) 2014-08-08T10:40:43.250 INFO:tasks.rados.rados.0.vpm167.stdout:3134: write oid 36 current snap is 415 2014-08-08T10:40:43.254 INFO:tasks.rados.rados.0.vpm167.stdout:3134: seq_num 1182 ranges {400529=697469,1617658=481285,2679005=121524} 2014-08-08T10:40:43.289 INFO:tasks.rados.rados.0.vpm167.stdout:3134: writing vpm1675861-36 from 400529 to 1097998 tid 1 2014-08-08T10:40:43.328 INFO:tasks.rados.rados.0.vpm167.stdout:3134: writing vpm1675861-36 from 1617658 to 2098943 tid 2 2014-08-08T10:40:43.337 INFO:tasks.rados.rados.0.vpm167.stdout:3134: writing vpm1675861-36 from 2679005 to 2800529 tid 3 2014-08-08T10:40:43.339 INFO:tasks.rados.rados.0.vpm167.stdout:3135: write oid 49 current snap is 415 2014-08-08T10:40:43.339 INFO:tasks.rados.rados.0.vpm167.stdout:3135: seq_num 1183 ranges {494735=401497,1425870=551480,2457958=36777} 2014-08-08T10:40:43.356 INFO:tasks.rados.rados.0.vpm167.stdout:3135: writing vpm1675861-49 from 494735 to 896232 tid 1 2014-08-08T10:40:43.380 INFO:tasks.rados.rados.0.vpm167.stdout:3135: writing vpm1675861-49 from 1425870 to 1977350 tid 2 2014-08-08T10:40:43.387 INFO:tasks.rados.rados.0.vpm167.stdout:3135: writing vpm1675861-49 from 2457958 to 2494735 tid 3 2014-08-08T10:40:43.388 INFO:tasks.rados.rados.0.vpm167.stdout:3136: write oid 45 current snap is 415 2014-08-08T10:40:43.388 INFO:tasks.rados.rados.0.vpm167.stdout:3136: seq_num 1184 ranges {439564=524725,1540001=621675,2660206=559911,3239563=1} 2014-08-08T10:40:43.408 INFO:tasks.rados.rados.0.vpm167.stdout:3136: writing vpm1675861-45 from 439564 to 964289 tid 1 2014-08-08T10:40:43.441 INFO:tasks.rados.rados.0.vpm167.stdout:3136: writing vpm1675861-45 from 1540001 to 2161676 tid 2 2014-08-08T10:40:43.469 INFO:tasks.rados.rados.0.vpm167.stdout:3136: writing vpm1675861-45 from 2660206 to 3220117 tid 3 2014-08-08T10:40:43.471 INFO:tasks.rados.rados.0.vpm167.stdout:3136: writing vpm1675861-45 from 3239563 to 3239564 tid 4 2014-08-08T10:40:43.471 INFO:tasks.rados.rados.0.vpm167.stdout:3135: finishing write tid 1 to vpm1675861-49 2014-08-08T10:40:43.471 INFO:tasks.rados.rados.0.vpm167.stdout:3135: finishing write tid 2 to vpm1675861-49 2014-08-08T10:40:43.472 INFO:tasks.rados.rados.0.vpm167.stdout:3135: finishing write tid 3 to vpm1675861-49 2014-08-08T10:40:43.472 INFO:tasks.rados.rados.0.vpm167.stdout:3135: finishing write tid 5 to vpm1675861-49 2014-08-08T10:40:43.472 INFO:tasks.rados.rados.0.vpm167.stdout:3137: delete oid 16 current snap is 415 2014-08-08T10:40:43.521 INFO:tasks.rados.rados.0.vpm167.stdout:3135: finishing write tid 6 to vpm1675861-49 2014-08-08T10:40:43.521 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 49 v 1233 (ObjNum 1183 snap 415 seq_num 1183) dirty exists 2014-08-08T10:40:43.702 INFO:tasks.rados.rados.0.vpm167.stdout:3134: finishing write tid 1 to vpm1675861-36 2014-08-08T10:40:43.732 INFO:tasks.rados.rados.0.vpm167.stdout:3133: finishing write tid 1 to vpm1675861-1 2014-08-08T10:40:43.732 INFO:tasks.rados.rados.0.vpm167.stdout:3133: finishing write tid 2 to vpm1675861-1 2014-08-08T10:40:43.757 INFO:tasks.rados.rados.0.vpm167.stdout:3132: expect (ObjNum 1060 snap 364 seq_num 1060) 2014-08-08T10:40:44.500 INFO:tasks.rados.rados.0.vpm167.stdout:3132: done (5 left) 2014-08-08T10:40:44.500 INFO:tasks.rados.rados.0.vpm167.stdout:3135: done (4 left) 2014-08-08T10:40:44.501 INFO:tasks.rados.rados.0.vpm167.stdout:3137: done (3 left) 2014-08-08T10:40:44.501 INFO:tasks.rados.rados.0.vpm167.stdout:3138: rollback oid 8 current snap is 415 2014-08-08T10:40:44.501 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 8 to 376 2014-08-08T10:40:44.501 INFO:tasks.rados.rados.0.vpm167.stdout:3139: delete oid 13 current snap is 415 2014-08-08T10:40:44.501 INFO:tasks.rados.rados.0.vpm167.stdout:3133: finishing write tid 4 to vpm1675861-1 2014-08-08T10:40:44.502 INFO:tasks.rados.rados.0.vpm167.stdout:3133: finishing write tid 5 to vpm1675861-1 2014-08-08T10:40:44.502 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 1 v 1305 (ObjNum 1181 snap 415 seq_num 1181) dirty exists 2014-08-08T10:40:44.502 INFO:tasks.rados.rados.0.vpm167.stdout:3134: finishing write tid 2 to vpm1675861-36 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 1 to vpm1675861-45 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 2 to vpm1675861-45 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:3134: finishing write tid 3 to vpm1675861-36 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:3134: finishing write tid 5 to vpm1675861-36 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:3134: finishing write tid 6 to vpm1675861-36 2014-08-08T10:40:44.503 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 36 v 1970 (ObjNum 1182 snap 415 seq_num 1182) dirty exists 2014-08-08T10:40:44.504 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 3 to vpm1675861-45 2014-08-08T10:40:44.504 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 4 to vpm1675861-45 2014-08-08T10:40:44.504 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 6 to vpm1675861-45 2014-08-08T10:40:44.504 INFO:tasks.rados.rados.0.vpm167.stdout:3136: finishing write tid 7 to vpm1675861-45 2014-08-08T10:40:44.504 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 1390 (ObjNum 1184 snap 415 seq_num 1184) dirty exists 2014-08-08T10:40:44.543 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 8 v 2083 (ObjNum 1063 snap 368 seq_num 1063) dirty exists 2014-08-08T10:40:44.722 INFO:tasks.rados.rados.0.vpm167.stdout:3133: done (4 left) 2014-08-08T10:40:44.722 INFO:tasks.rados.rados.0.vpm167.stdout:3134: done (3 left) 2014-08-08T10:40:44.723 INFO:tasks.rados.rados.0.vpm167.stdout:3136: done (2 left) 2014-08-08T10:40:44.724 INFO:tasks.rados.rados.0.vpm167.stdout:3138: done (1 left) 2014-08-08T10:40:44.724 INFO:tasks.rados.rados.0.vpm167.stdout:3139: done (0 left) 2014-08-08T10:40:44.724 INFO:tasks.rados.rados.0.vpm167.stdout:3140: read oid 40 snap 389 2014-08-08T10:40:44.724 INFO:tasks.rados.rados.0.vpm167.stdout:3141: rollback oid 45 current snap is 415 2014-08-08T10:40:44.725 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 45 to 397 2014-08-08T10:40:44.725 INFO:tasks.rados.rados.0.vpm167.stdout:3142: write oid 37 current snap is 415 2014-08-08T10:40:44.725 INFO:tasks.rados.rados.0.vpm167.stdout:3142: seq_num 1187 ranges {540806=400000} 2014-08-08T10:40:44.766 INFO:tasks.rados.rados.0.vpm167.stdout:3142: writing vpm1675861-37 from 540806 to 940806 tid 1 2014-08-08T10:40:44.770 INFO:tasks.rados.rados.0.vpm167.stdout:3140: expect (ObjNum 990 snap 335 seq_num 990) 2014-08-08T10:40:45.848 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5] out_osds: [2] dead_osds: [3] live_osds: [1, 0, 5, 4, 2] 2014-08-08T10:40:45.849 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:45.849 INFO:tasks.thrashosds.thrasher:Adding osd 2 2014-08-08T10:40:45.849 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-08-08T10:40:46.614 INFO:teuthology.orchestra.run.vpm166.stderr:marked in osd.2. 2014-08-08T10:40:46.663 INFO:tasks.thrashosds.thrasher:Added osd 2 2014-08-08T10:40:47.199 INFO:tasks.rados.rados.0.vpm167.stdout:3140: done (2 left) 2014-08-08T10:40:47.199 INFO:tasks.rados.rados.0.vpm167.stdout:3143: rollback oid 7 current snap is 415 2014-08-08T10:40:47.199 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 7 to 396 2014-08-08T10:40:47.200 INFO:tasks.rados.rados.0.vpm167.stdout:3144: write oid 47 current snap is 415 2014-08-08T10:40:47.200 INFO:tasks.rados.rados.0.vpm167.stdout:3144: seq_num 1188 ranges {484611=599463,1525362=159249} 2014-08-08T10:40:47.223 INFO:tasks.rados.rados.0.vpm167.stdout:3144: writing vpm1675861-47 from 484611 to 1084074 tid 1 2014-08-08T10:40:47.234 INFO:tasks.rados.rados.0.vpm167.stdout:3144: writing vpm1675861-47 from 1525362 to 1684611 tid 2 2014-08-08T10:40:47.235 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 1392 (ObjNum -1 snap 0 seq_num 0) dirty dne 2014-08-08T10:40:47.236 INFO:tasks.rados.rados.0.vpm167.stdout:3142: finishing write tid 1 to vpm1675861-37 2014-08-08T10:40:47.236 INFO:tasks.rados.rados.0.vpm167.stdout:3142: finishing write tid 3 to vpm1675861-37 2014-08-08T10:40:47.236 INFO:tasks.rados.rados.0.vpm167.stdout:3142: finishing write tid 4 to vpm1675861-37 2014-08-08T10:40:47.236 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 37 v 1972 (ObjNum 1187 snap 415 seq_num 1187) dirty exists 2014-08-08T10:40:47.236 INFO:tasks.rados.rados.0.vpm167.stdout:3141: done (3 left) 2014-08-08T10:40:47.237 INFO:tasks.rados.rados.0.vpm167.stdout:3142: done (2 left) 2014-08-08T10:40:47.237 INFO:tasks.rados.rados.0.vpm167.stdout:3145: snap_remove snap 406 2014-08-08T10:40:47.416 INFO:tasks.rados.rados.0.vpm167.stdout:3144: finishing write tid 1 to vpm1675861-47 2014-08-08T10:40:47.416 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 770 (ObjNum 1083 snap 376 seq_num 1083) dirty exists 2014-08-08T10:40:47.417 INFO:tasks.rados.rados.0.vpm167.stdout:3144: finishing write tid 2 to vpm1675861-47 2014-08-08T10:40:47.417 INFO:tasks.rados.rados.0.vpm167.stdout:3144: finishing write tid 4 to vpm1675861-47 2014-08-08T10:40:47.575 INFO:tasks.rados.rados.0.vpm167.stdout:3144: finishing write tid 5 to vpm1675861-47 2014-08-08T10:40:47.575 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 47 v 1309 (ObjNum 1188 snap 415 seq_num 1188) dirty exists 2014-08-08T10:40:47.902 INFO:tasks.rados.rados.0.vpm167.stdout:3143: done (2 left) 2014-08-08T10:40:47.902 INFO:tasks.rados.rados.0.vpm167.stdout:3144: done (1 left) 2014-08-08T10:40:47.902 INFO:tasks.rados.rados.0.vpm167.stdout:3145: done (0 left) 2014-08-08T10:40:47.902 INFO:tasks.rados.rados.0.vpm167.stdout:3146: rollback oid 16 current snap is 415 2014-08-08T10:40:47.903 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 16 to 399 2014-08-08T10:40:47.903 INFO:tasks.rados.rados.0.vpm167.stdout:3147: rollback oid 5 current snap is 415 2014-08-08T10:40:47.903 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 5 to 387 2014-08-08T10:40:47.904 INFO:tasks.rados.rados.0.vpm167.stdout:3148: read oid 33 snap 396 2014-08-08T10:40:47.904 INFO:tasks.rados.rados.0.vpm167.stdout:3149: read oid 49 snap -1 2014-08-08T10:40:47.905 INFO:tasks.rados.rados.0.vpm167.stdout:3150: delete oid 25 current snap is 415 2014-08-08T10:40:47.921 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 5 v 1134 (ObjNum 1079 snap 374 seq_num 1079) dirty exists 2014-08-08T10:40:47.942 INFO:tasks.rados.rados.0.vpm167.stdout:3149: expect (ObjNum 1183 snap 415 seq_num 1183) 2014-08-08T10:40:49.204 INFO:tasks.rados.rados.0.vpm167.stdout:3148: expect (ObjNum 889 snap 301 seq_num 889) 2014-08-08T10:40:51.311 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1394 (ObjNum 1106 snap 391 seq_num 1106) dirty exists 2014-08-08T10:40:51.663 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5, 2] out_osds: [] dead_osds: [3] live_osds: [1, 0, 5, 4, 2] 2014-08-08T10:40:51.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:51.663 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-08T10:40:51.663 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:40:51.663 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:40:51.663 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-08T10:40:53.147 INFO:tasks.rados.rados.0.vpm167.stdout:3146: done (4 left) 2014-08-08T10:40:53.148 INFO:tasks.rados.rados.0.vpm167.stdout:3147: done (3 left) 2014-08-08T10:40:53.148 INFO:tasks.rados.rados.0.vpm167.stdout:3148: done (2 left) 2014-08-08T10:40:53.148 INFO:tasks.rados.rados.0.vpm167.stdout:3149: done (1 left) 2014-08-08T10:40:53.148 INFO:tasks.rados.rados.0.vpm167.stdout:3150: done (0 left) 2014-08-08T10:40:53.148 INFO:tasks.rados.rados.0.vpm167.stdout:3151: read oid 26 snap -1 2014-08-08T10:40:53.149 INFO:tasks.rados.rados.0.vpm167.stdout:3152: read oid 47 snap -1 2014-08-08T10:40:53.149 INFO:tasks.rados.rados.0.vpm167.stdout:3153: read oid 2 snap -1 2014-08-08T10:40:53.149 INFO:tasks.rados.rados.0.vpm167.stdout:3154: read oid 28 snap -1 2014-08-08T10:40:53.150 INFO:tasks.rados.rados.0.vpm167.stdout:3155: delete oid 50 current snap is 415 2014-08-08T10:40:53.165 INFO:tasks.rados.rados.0.vpm167.stdout:3151: expect (ObjNum 1071 snap 371 seq_num 1071) 2014-08-08T10:40:53.186 INFO:tasks.rados.rados.0.vpm167.stdout:3152: expect (ObjNum 1188 snap 415 seq_num 1188) 2014-08-08T10:40:53.745 INFO:tasks.rados.rados.0.vpm167.stdout:3151: done (4 left) 2014-08-08T10:40:53.745 INFO:tasks.rados.rados.0.vpm167.stdout:3152: done (3 left) 2014-08-08T10:40:53.745 INFO:tasks.rados.rados.0.vpm167.stdout:3155: done (2 left) 2014-08-08T10:40:53.746 INFO:tasks.rados.rados.0.vpm167.stdout:3156: rollback oid 32 current snap is 415 2014-08-08T10:40:53.746 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 32 to 414 2014-08-08T10:40:53.747 INFO:tasks.rados.rados.0.vpm167.stdout:3157: rollback oid 36 current snap is 415 2014-08-08T10:40:53.747 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 36 to 370 2014-08-08T10:40:53.747 INFO:tasks.rados.rados.0.vpm167.stdout:3158: read oid 26 snap -1 2014-08-08T10:40:53.747 INFO:tasks.rados.rados.0.vpm167.stdout:3159: write oid 10 current snap is 415 2014-08-08T10:40:53.747 INFO:tasks.rados.rados.0.vpm167.stdout:3159: seq_num 1191 ranges {585853=550972,1680239=499537,2663089=605205,3385852=1} 2014-08-08T10:40:53.769 INFO:tasks.rados.rados.0.vpm167.stdout:3159: writing vpm1675861-10 from 585853 to 1136825 tid 1 2014-08-08T10:40:53.791 INFO:tasks.rados.rados.0.vpm167.stdout:3159: writing vpm1675861-10 from 1680239 to 2179776 tid 2 2014-08-08T10:40:53.816 INFO:tasks.rados.rados.0.vpm167.stdout:3159: writing vpm1675861-10 from 2663089 to 3268294 tid 3 2014-08-08T10:40:53.817 INFO:tasks.rados.rados.0.vpm167.stdout:3159: writing vpm1675861-10 from 3385852 to 3385853 tid 4 2014-08-08T10:40:53.818 INFO:tasks.rados.rados.0.vpm167.stdout:3153: expect (ObjNum 1101 snap 385 seq_num 1101) 2014-08-08T10:40:54.804 INFO:tasks.rados.rados.0.vpm167.stdout:3153: done (5 left) 2014-08-08T10:40:54.805 INFO:tasks.rados.rados.0.vpm167.stdout:3154: done (4 left) 2014-08-08T10:40:54.805 INFO:tasks.rados.rados.0.vpm167.stdout:3160: read oid 37 snap -1 2014-08-08T10:40:54.805 INFO:tasks.rados.rados.0.vpm167.stdout:3161: read oid 34 snap -1 2014-08-08T10:40:54.805 INFO:tasks.rados.rados.0.vpm167.stdout:3162: snap_create 2014-08-08T10:40:54.806 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 32 v 0 (ObjNum -1 snap 0 seq_num 0) dirty dne 2014-08-08T10:40:54.806 INFO:tasks.rados.rados.0.vpm167.stdout:3158: expect (ObjNum 1071 snap 371 seq_num 1071) 2014-08-08T10:40:54.825 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 36 v 1979 (ObjNum 1039 snap 356 seq_num 1039) dirty exists 2014-08-08T10:40:54.826 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 1 to vpm1675861-10 2014-08-08T10:40:54.826 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 2 to vpm1675861-10 2014-08-08T10:40:54.826 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 3 to vpm1675861-10 2014-08-08T10:40:54.826 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 4 to vpm1675861-10 2014-08-08T10:40:54.827 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 6 to vpm1675861-10 2014-08-08T10:40:54.827 INFO:tasks.rados.rados.0.vpm167.stdout:3159: finishing write tid 7 to vpm1675861-10 2014-08-08T10:40:54.827 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 10 v 903 (ObjNum 1191 snap 415 seq_num 1191) dirty exists 2014-08-08T10:40:54.827 INFO:tasks.rados.rados.0.vpm167.stdout:3160: expect (ObjNum 1187 snap 415 seq_num 1187) 2014-08-08T10:40:55.512 INFO:tasks.rados.rados.0.vpm167.stdout:3156: done (6 left) 2014-08-08T10:40:55.512 INFO:tasks.rados.rados.0.vpm167.stdout:3157: done (5 left) 2014-08-08T10:40:55.512 INFO:tasks.rados.rados.0.vpm167.stdout:3158: done (4 left) 2014-08-08T10:40:55.513 INFO:tasks.rados.rados.0.vpm167.stdout:3159: done (3 left) 2014-08-08T10:40:55.513 INFO:tasks.rados.rados.0.vpm167.stdout:3160: done (2 left) 2014-08-08T10:40:55.513 INFO:tasks.rados.rados.0.vpm167.stdout:3161: done (1 left) 2014-08-08T10:40:55.513 INFO:tasks.rados.rados.0.vpm167.stdout:3162: done (0 left) 2014-08-08T10:40:55.513 INFO:tasks.rados.rados.0.vpm167.stdout:3163: read oid 36 snap 358 2014-08-08T10:40:55.514 INFO:tasks.rados.rados.0.vpm167.stdout:3164: rollback oid 3 current snap is 416 2014-08-08T10:40:55.514 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 388 2014-08-08T10:40:55.515 INFO:tasks.rados.rados.0.vpm167.stdout:3165: write oid 11 current snap is 416 2014-08-08T10:40:55.515 INFO:tasks.rados.rados.0.vpm167.stdout:3165: seq_num 1192 ranges {681082=674200,1944770=720133,3081081=1} 2014-08-08T10:40:55.558 INFO:tasks.rados.rados.0.vpm167.stdout:3165: writing vpm1675861-11 from 681082 to 1355282 tid 1 2014-08-08T10:40:55.605 INFO:tasks.rados.rados.0.vpm167.stdout:3165: writing vpm1675861-11 from 1944770 to 2664903 tid 2 2014-08-08T10:40:55.611 INFO:tasks.rados.rados.0.vpm167.stdout:3165: writing vpm1675861-11 from 3081081 to 3081082 tid 3 2014-08-08T10:40:55.612 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 0 (ObjNum -1017313344 snap 32533 seq_num 1703937) dirty dne 2014-08-08T10:40:55.612 INFO:tasks.rados.rados.0.vpm167.stdout:3163: expect (ObjNum 1039 snap 356 seq_num 1039) 2014-08-08T10:40:55.645 INFO:tasks.rados.rados.0.vpm167.stdout:3163: done (2 left) 2014-08-08T10:40:55.645 INFO:tasks.rados.rados.0.vpm167.stdout:3164: done (1 left) 2014-08-08T10:40:55.645 INFO:tasks.rados.rados.0.vpm167.stdout:3166: snap_create 2014-08-08T10:40:55.646 INFO:tasks.rados.rados.0.vpm167.stdout:3165: finishing write tid 1 to vpm1675861-11 2014-08-08T10:40:55.773 INFO:tasks.rados.rados.0.vpm167.stdout:3165: finishing write tid 2 to vpm1675861-11 2014-08-08T10:40:55.782 INFO:tasks.rados.rados.0.vpm167.stdout:3165: finishing write tid 3 to vpm1675861-11 2014-08-08T10:40:55.792 INFO:tasks.rados.rados.0.vpm167.stdout:3165: finishing write tid 5 to vpm1675861-11 2014-08-08T10:40:55.806 INFO:tasks.rados.rados.0.vpm167.stdout:3165: finishing write tid 6 to vpm1675861-11 2014-08-08T10:40:55.807 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 1043 (ObjNum 1192 snap 416 seq_num 1192) dirty exists 2014-08-08T10:40:56.630 INFO:tasks.rados.rados.0.vpm167.stdout:3165: done (1 left) 2014-08-08T10:40:56.630 INFO:tasks.rados.rados.0.vpm167.stdout:3166: done (0 left) 2014-08-08T10:40:56.630 INFO:tasks.rados.rados.0.vpm167.stdout:3167: snap_create 2014-08-08T10:40:56.762 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5, 2] out_osds: [] dead_osds: [3] live_osds: [1, 0, 5, 4, 2] 2014-08-08T10:40:56.762 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:40:56.762 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2014-08-08T10:40:56.762 INFO:tasks.ceph.osd.3:Restarting daemon 2014-08-08T10:40:56.763 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-08-08T10:40:56.766 INFO:tasks.ceph.osd.3:Started 2014-08-08T10:40:56.766 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-08-08T10:40:56.889 INFO:teuthology.orchestra.run.vpm165.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2014-08-08T10:40:56.896 INFO:tasks.thrashosds.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2014-08-08T10:40:57.250 INFO:tasks.ceph.osd.3.vpm165.stdout:starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-08-08T10:40:57.465 INFO:tasks.ceph.osd.3.vpm165.stderr:2014-08-08 17:40:57.463845 7ff22e0b68c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-08-08T10:40:57.669 INFO:tasks.rados.rados.0.vpm167.stdout:3167: done (0 left) 2014-08-08T10:40:57.669 INFO:tasks.rados.rados.0.vpm167.stdout:3168: snap_remove snap 376 2014-08-08T10:40:58.916 INFO:tasks.rados.rados.0.vpm167.stdout:3168: done (0 left) 2014-08-08T10:40:58.916 INFO:tasks.rados.rados.0.vpm167.stdout:3169: snap_remove snap 411 2014-08-08T10:41:00.012 INFO:tasks.rados.rados.0.vpm167.stdout:3169: done (0 left) 2014-08-08T10:41:00.012 INFO:tasks.rados.rados.0.vpm167.stdout:3170: snap_create 2014-08-08T10:41:01.200 INFO:tasks.rados.rados.0.vpm167.stdout:3170: done (0 left) 2014-08-08T10:41:01.200 INFO:tasks.rados.rados.0.vpm167.stdout:3171: snap_remove snap 414 2014-08-08T10:41:01.896 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-08-08T10:41:02.526 INFO:tasks.rados.rados.0.vpm167.stdout:3171: done (0 left) 2014-08-08T10:41:02.526 INFO:tasks.rados.rados.0.vpm167.stdout:3172: write oid 50 current snap is 419 2014-08-08T10:41:02.526 INFO:tasks.rados.rados.0.vpm167.stdout:3172: seq_num 1193 ranges {624888=648053,1873431=790328,3066564=625314,3824887=1} 2014-08-08T10:41:02.558 INFO:tasks.rados.rados.0.vpm167.stdout:3172: writing vpm1675861-50 from 624888 to 1272941 tid 1 2014-08-08T10:41:02.609 INFO:tasks.rados.rados.0.vpm167.stdout:3172: writing vpm1675861-50 from 1873431 to 2663759 tid 2 2014-08-08T10:41:02.649 INFO:tasks.rados.rados.0.vpm167.stdout:3172: writing vpm1675861-50 from 3066564 to 3691878 tid 3 2014-08-08T10:41:02.650 INFO:tasks.rados.rados.0.vpm167.stdout:3172: writing vpm1675861-50 from 3824887 to 3824888 tid 4 2014-08-08T10:41:02.651 INFO:tasks.rados.rados.0.vpm167.stdout:3173: read oid 4 snap -1 2014-08-08T10:41:02.651 INFO:tasks.rados.rados.0.vpm167.stdout:3174: delete oid 14 current snap is 419 2014-08-08T10:41:03.008 INFO:tasks.rados.rados.0.vpm167.stdout:3174: done (2 left) 2014-08-08T10:41:03.008 INFO:tasks.rados.rados.0.vpm167.stdout:3175: rollback oid 12 current snap is 419 2014-08-08T10:41:03.009 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 12 to 355 2014-08-08T10:41:03.009 INFO:tasks.rados.rados.0.vpm167.stdout:3176: read oid 27 snap 369 2014-08-08T10:41:03.010 INFO:tasks.rados.rados.0.vpm167.stdout:3177: read oid 14 snap -1 2014-08-08T10:41:03.010 INFO:tasks.rados.rados.0.vpm167.stdout:3178: read oid 31 snap -1 2014-08-08T10:41:03.011 INFO:tasks.rados.rados.0.vpm167.stdout:3179: write oid 22 current snap is 419 2014-08-08T10:41:03.012 INFO:tasks.rados.rados.0.vpm167.stdout:3179: seq_num 1195 ranges {781299=1} 2014-08-08T10:41:03.021 INFO:tasks.rados.rados.0.vpm167.stdout:3179: writing vpm1675861-22 from 781299 to 781300 tid 1 2014-08-08T10:41:03.023 INFO:tasks.rados.rados.0.vpm167.stdout:3177: done (6 left) 2014-08-08T10:41:03.023 INFO:tasks.rados.rados.0.vpm167.stdout:3180: write oid 21 current snap is 419 2014-08-08T10:41:03.023 INFO:tasks.rados.rados.0.vpm167.stdout:3180: seq_num 1196 ranges {726129=599562,1526128=1} 2014-08-08T10:41:03.059 INFO:tasks.rados.rados.0.vpm167.stdout:3180: writing vpm1675861-21 from 726129 to 1325691 tid 1 2014-08-08T10:41:03.067 INFO:tasks.rados.rados.0.vpm167.stdout:3180: writing vpm1675861-21 from 1526128 to 1526129 tid 2 2014-08-08T10:41:03.069 INFO:tasks.rados.rados.0.vpm167.stdout:3181: write oid 19 current snap is 419 2014-08-08T10:41:03.069 INFO:tasks.rados.rados.0.vpm167.stdout:3181: seq_num 1197 ranges {422383=571366,1222382=1} 2014-08-08T10:41:03.102 INFO:tasks.rados.rados.0.vpm167.stdout:3181: writing vpm1675861-19 from 422383 to 993749 tid 1 2014-08-08T10:41:03.105 INFO:tasks.rados.rados.0.vpm167.stdout:3181: writing vpm1675861-19 from 1222382 to 1222383 tid 2 2014-08-08T10:41:03.105 INFO:tasks.rados.rados.0.vpm167.stdout:3182: snap_create 2014-08-08T10:41:03.158 INFO:tasks.rados.rados.0.vpm167.stdout:3180: finishing write tid 1 to vpm1675861-21 2014-08-08T10:41:03.188 INFO:tasks.rados.rados.0.vpm167.stdout:3180: finishing write tid 2 to vpm1675861-21 2014-08-08T10:41:03.202 INFO:tasks.rados.rados.0.vpm167.stdout:3180: finishing write tid 4 to vpm1675861-21 2014-08-08T10:41:03.433 INFO:tasks.rados.rados.0.vpm167.stdout:3178: expect (ObjNum 1173 snap 413 seq_num 1173) 2014-08-08T10:41:04.802 INFO:tasks.rados.rados.0.vpm167.stdout:3178: done (8 left) 2014-08-08T10:41:04.802 INFO:tasks.rados.rados.0.vpm167.stdout:3182: done (7 left) 2014-08-08T10:41:04.803 INFO:tasks.rados.rados.0.vpm167.stdout:3183: write oid 26 current snap is 420 2014-08-08T10:41:04.803 INFO:tasks.rados.rados.0.vpm167.stdout:3183: seq_num 1198 ranges {766189=694594,1966188=1} 2014-08-08T10:41:04.830 INFO:tasks.rados.rados.0.vpm167.stdout:3183: writing vpm1675861-26 from 766189 to 1460783 tid 1 2014-08-08T10:41:04.839 INFO:tasks.rados.rados.0.vpm167.stdout:3183: writing vpm1675861-26 from 1966188 to 1966189 tid 2 2014-08-08T10:41:04.840 INFO:tasks.rados.rados.0.vpm167.stdout:3180: finishing write tid 5 to vpm1675861-21 2014-08-08T10:41:04.840 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 21 v 394 (ObjNum 1196 snap 419 seq_num 1196) dirty exists 2014-08-08T10:41:04.840 INFO:tasks.rados.rados.0.vpm167.stdout:3173: expect (ObjNum 1145 snap 405 seq_num 1145) 2014-08-08T10:41:06.329 INFO:tasks.rados.rados.0.vpm167.stdout:3173: done (7 left) 2014-08-08T10:41:06.330 INFO:tasks.rados.rados.0.vpm167.stdout:3180: done (6 left) 2014-08-08T10:41:06.330 INFO:tasks.rados.rados.0.vpm167.stdout:3184: write oid 11 current snap is 420 2014-08-08T10:41:06.330 INFO:tasks.rados.rados.0.vpm167.stdout:3184: seq_num 1199 ranges {711018=550047,1666832=597995,2711017=1} 2014-08-08T10:41:06.364 INFO:tasks.rados.rados.0.vpm167.stdout:3184: writing vpm1675861-11 from 711018 to 1261065 tid 1 2014-08-08T10:41:06.400 INFO:tasks.rados.rados.0.vpm167.stdout:3184: writing vpm1675861-11 from 1666832 to 2264827 tid 2 2014-08-08T10:41:06.407 INFO:tasks.rados.rados.0.vpm167.stdout:3184: writing vpm1675861-11 from 2711017 to 2711018 tid 3 2014-08-08T10:41:06.408 INFO:tasks.rados.rados.0.vpm167.stdout:3176: expect (ObjNum 1029 snap 353 seq_num 1029) 2014-08-08T10:41:07.179 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 0, 5, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:07.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:07.179 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 4, 3, 0, 5, 2] 2014-08-08T10:41:07.179 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-08T10:41:09.984 INFO:tasks.rados.rados.0.vpm167.stdout:3176: done (6 left) 2014-08-08T10:41:09.985 INFO:tasks.rados.rados.0.vpm167.stdout:3185: snap_remove snap 418 2014-08-08T10:41:09.985 INFO:tasks.rados.rados.0.vpm167.stdout:3179: finishing write tid 1 to vpm1675861-22 2014-08-08T10:41:09.986 INFO:tasks.rados.rados.0.vpm167.stdout:3179: finishing write tid 3 to vpm1675861-22 2014-08-08T10:41:09.986 INFO:tasks.rados.rados.0.vpm167.stdout:3179: finishing write tid 4 to vpm1675861-22 2014-08-08T10:41:09.986 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 22 v 1236 (ObjNum 1195 snap 419 seq_num 1195) dirty exists 2014-08-08T10:41:09.987 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2085 (ObjNum 950 snap 312 seq_num 950) dirty exists 2014-08-08T10:41:09.987 INFO:tasks.rados.rados.0.vpm167.stdout:3183: finishing write tid 1 to vpm1675861-26 2014-08-08T10:41:09.988 INFO:tasks.rados.rados.0.vpm167.stdout:3183: finishing write tid 2 to vpm1675861-26 2014-08-08T10:41:09.988 INFO:tasks.rados.rados.0.vpm167.stdout:3183: finishing write tid 4 to vpm1675861-26 2014-08-08T10:41:09.988 INFO:tasks.rados.rados.0.vpm167.stdout:3183: finishing write tid 5 to vpm1675861-26 2014-08-08T10:41:09.988 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 863 (ObjNum 1198 snap 420 seq_num 1198) dirty exists 2014-08-08T10:41:09.989 INFO:tasks.rados.rados.0.vpm167.stdout:3181: finishing write tid 1 to vpm1675861-19 2014-08-08T10:41:09.989 INFO:tasks.rados.rados.0.vpm167.stdout:3181: finishing write tid 2 to vpm1675861-19 2014-08-08T10:41:09.989 INFO:tasks.rados.rados.0.vpm167.stdout:3184: finishing write tid 1 to vpm1675861-11 2014-08-08T10:41:09.989 INFO:tasks.rados.rados.0.vpm167.stdout:3184: finishing write tid 2 to vpm1675861-11 2014-08-08T10:41:09.991 INFO:tasks.rados.rados.0.vpm167.stdout:3184: finishing write tid 3 to vpm1675861-11 2014-08-08T10:41:09.991 INFO:tasks.rados.rados.0.vpm167.stdout:3184: finishing write tid 5 to vpm1675861-11 2014-08-08T10:41:09.991 INFO:tasks.rados.rados.0.vpm167.stdout:3181: finishing write tid 4 to vpm1675861-19 2014-08-08T10:41:09.991 INFO:tasks.rados.rados.0.vpm167.stdout:3184: finishing write tid 6 to vpm1675861-11 2014-08-08T10:41:09.991 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 1060 (ObjNum 1199 snap 420 seq_num 1199) dirty exists 2014-08-08T10:41:09.992 INFO:tasks.rados.rados.0.vpm167.stdout:3181: finishing write tid 5 to vpm1675861-19 2014-08-08T10:41:09.992 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 19 v 1313 (ObjNum 1197 snap 419 seq_num 1197) dirty exists 2014-08-08T10:41:09.992 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 1 to vpm1675861-50 2014-08-08T10:41:09.992 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 2 to vpm1675861-50 2014-08-08T10:41:09.993 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 3 to vpm1675861-50 2014-08-08T10:41:09.993 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 4 to vpm1675861-50 2014-08-08T10:41:09.993 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 6 to vpm1675861-50 2014-08-08T10:41:09.993 INFO:tasks.rados.rados.0.vpm167.stdout:3172: finishing write tid 7 to vpm1675861-50 2014-08-08T10:41:09.993 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 50 v 729 (ObjNum 1193 snap 419 seq_num 1193) dirty exists 2014-08-08T10:41:10.314 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.0. 2014-08-08T10:41:11.404 INFO:tasks.rados.rados.0.vpm167.stdout:3172: done (6 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3175: done (5 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3179: done (4 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3181: done (3 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3183: done (2 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3184: done (1 left) 2014-08-08T10:41:11.405 INFO:tasks.rados.rados.0.vpm167.stdout:3185: done (0 left) 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:3186: rollback oid 16 current snap is 420 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 16 to 369 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:3187: rollback oid 43 current snap is 420 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 43 to 384 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:3188: rollback oid 6 current snap is 420 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 6 to 397 2014-08-08T10:41:11.406 INFO:tasks.rados.rados.0.vpm167.stdout:3189: read oid 8 snap 355 2014-08-08T10:41:11.407 INFO:tasks.rados.rados.0.vpm167.stdout:3190: snap_create 2014-08-08T10:41:11.982 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 6 v 1694 (ObjNum 1112 snap 392 seq_num 1112) dirty exists 2014-08-08T10:41:12.492 INFO:tasks.rados.rados.0.vpm167.stdout:3188: done (4 left) 2014-08-08T10:41:12.494 INFO:tasks.rados.rados.0.vpm167.stdout:3189: done (3 left) 2014-08-08T10:41:12.494 INFO:tasks.rados.rados.0.vpm167.stdout:3190: done (2 left) 2014-08-08T10:41:12.494 INFO:tasks.rados.rados.0.vpm167.stdout:3191: write oid 9 current snap is 421 2014-08-08T10:41:12.494 INFO:tasks.rados.rados.0.vpm167.stdout:3191: seq_num 1200 ranges {521576=674299,1764068=419614,2921575=1} 2014-08-08T10:41:12.534 INFO:tasks.rados.rados.0.vpm167.stdout:3191: writing vpm1675861-9 from 521576 to 1195875 tid 1 2014-08-08T10:41:12.564 INFO:tasks.rados.rados.0.vpm167.stdout:3191: writing vpm1675861-9 from 1764068 to 2183682 tid 2 2014-08-08T10:41:12.571 INFO:tasks.rados.rados.0.vpm167.stdout:3191: writing vpm1675861-9 from 2921575 to 2921576 tid 3 2014-08-08T10:41:12.572 INFO:tasks.rados.rados.0.vpm167.stdout:3192: read oid 10 snap 396 2014-08-08T10:41:12.573 INFO:tasks.rados.rados.0.vpm167.stdout:3193: rollback oid 23 current snap is 421 2014-08-08T10:41:12.573 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 23 to 386 2014-08-08T10:41:12.574 INFO:tasks.rados.rados.0.vpm167.stdout:3194: rollback oid 14 current snap is 421 2014-08-08T10:41:12.574 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 415 2014-08-08T10:41:12.574 INFO:tasks.rados.rados.0.vpm167.stdout:3195: delete oid 37 current snap is 421 2014-08-08T10:41:13.006 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 805 (ObjNum 57184040 snap 0 seq_num 30730944) dirty dne 2014-08-08T10:41:13.388 INFO:tasks.rados.rados.0.vpm167.stdout:3187: done (6 left) 2014-08-08T10:41:13.388 INFO:tasks.rados.rados.0.vpm167.stdout:3195: done (5 left) 2014-08-08T10:41:13.388 INFO:tasks.rados.rados.0.vpm167.stdout:3196: read oid 18 snap 413 2014-08-08T10:41:13.389 INFO:tasks.rados.rados.0.vpm167.stdout:3197: write oid 47 current snap is 421 2014-08-08T10:41:13.389 INFO:tasks.rados.rados.0.vpm167.stdout:3197: seq_num 1202 ranges {411235=1} 2014-08-08T10:41:13.392 INFO:tasks.rados.rados.0.vpm167.stdout:3197: writing vpm1675861-47 from 411235 to 411236 tid 1 2014-08-08T10:41:13.393 INFO:tasks.rados.rados.0.vpm167.stdout:3198: snap_remove snap 355 2014-08-08T10:41:13.394 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 1999 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:41:14.487 INFO:tasks.rados.rados.0.vpm167.stdout:3194: done (7 left) 2014-08-08T10:41:14.487 INFO:tasks.rados.rados.0.vpm167.stdout:3196: done (6 left) 2014-08-08T10:41:14.488 INFO:tasks.rados.rados.0.vpm167.stdout:3198: done (5 left) 2014-08-08T10:41:14.488 INFO:tasks.rados.rados.0.vpm167.stdout:3199: write oid 15 current snap is 421 2014-08-08T10:41:14.489 INFO:tasks.rados.rados.0.vpm167.stdout:3199: seq_num 1203 ranges {106464=1} 2014-08-08T10:41:14.489 INFO:tasks.rados.rados.0.vpm167.stdout:3199: writing vpm1675861-15 from 106464 to 106465 tid 1 2014-08-08T10:41:14.489 INFO:tasks.rados.rados.0.vpm167.stdout:3200: snap_remove snap 366 2014-08-08T10:41:15.035 INFO:tasks.rados.rados.0.vpm167.stdout:3197: finishing write tid 1 to vpm1675861-47 2014-08-08T10:41:15.386 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:15.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:15.386 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2014-08-08T10:41:15.386 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:41:15.387 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:41:15.387 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-08-08T10:41:15.961 INFO:tasks.rados.rados.0.vpm167.stdout:3200: done (6 left) 2014-08-08T10:41:15.961 INFO:tasks.rados.rados.0.vpm167.stdout:3201: snap_create 2014-08-08T10:41:17.069 INFO:tasks.rados.rados.0.vpm167.stdout:3201: done (6 left) 2014-08-08T10:41:17.069 INFO:tasks.rados.rados.0.vpm167.stdout:3202: delete oid 49 current snap is 422 2014-08-08T10:41:17.283 INFO:tasks.rados.rados.0.vpm167.stdout:3197: finishing write tid 3 to vpm1675861-47 2014-08-08T10:41:17.653 INFO:tasks.rados.rados.0.vpm167.stdout:3197: finishing write tid 4 to vpm1675861-47 2014-08-08T10:41:17.653 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 47 v 1316 (ObjNum 1202 snap 421 seq_num 1202) dirty exists 2014-08-08T10:41:17.724 INFO:tasks.rados.rados.0.vpm167.stdout:3192: done (6 left) 2014-08-08T10:41:17.725 INFO:tasks.rados.rados.0.vpm167.stdout:3197: done (5 left) 2014-08-08T10:41:17.725 INFO:tasks.rados.rados.0.vpm167.stdout:3202: done (4 left) 2014-08-08T10:41:17.725 INFO:tasks.rados.rados.0.vpm167.stdout:3203: read oid 36 snap 417 2014-08-08T10:41:17.725 INFO:tasks.rados.rados.0.vpm167.stdout:3204: read oid 41 snap -1 2014-08-08T10:41:17.726 INFO:tasks.rados.rados.0.vpm167.stdout:3205: read oid 14 snap -1 2014-08-08T10:41:17.726 INFO:tasks.rados.rados.0.vpm167.stdout:3206: snap_remove snap 371 2014-08-08T10:41:17.736 INFO:tasks.rados.rados.0.vpm167.stdout:3203: expect (ObjNum 1039 snap 356 seq_num 1039) 2014-08-08T10:41:17.772 INFO:tasks.rados.rados.0.vpm167.stdout:3205: expect (ObjNum 1172 snap 412 seq_num 1172) 2014-08-08T10:41:18.112 INFO:tasks.rados.rados.0.vpm167.stdout:3191: finishing write tid 1 to vpm1675861-9 2014-08-08T10:41:18.113 INFO:tasks.rados.rados.0.vpm167.stdout:3191: finishing write tid 2 to vpm1675861-9 2014-08-08T10:41:18.115 INFO:tasks.rados.rados.0.vpm167.stdout:3191: finishing write tid 3 to vpm1675861-9 2014-08-08T10:41:18.116 INFO:tasks.rados.rados.0.vpm167.stdout:3191: finishing write tid 5 to vpm1675861-9 2014-08-08T10:41:18.140 INFO:tasks.rados.rados.0.vpm167.stdout:3191: finishing write tid 6 to vpm1675861-9 2014-08-08T10:41:18.140 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 9 v 1703 (ObjNum 1200 snap 421 seq_num 1200) dirty exists 2014-08-08T10:41:18.563 INFO:tasks.rados.rados.0.vpm167.stdout:3191: done (7 left) 2014-08-08T10:41:18.563 INFO:tasks.rados.rados.0.vpm167.stdout:3203: done (6 left) 2014-08-08T10:41:18.563 INFO:tasks.rados.rados.0.vpm167.stdout:3204: done (5 left) 2014-08-08T10:41:18.564 INFO:tasks.rados.rados.0.vpm167.stdout:3205: done (4 left) 2014-08-08T10:41:18.564 INFO:tasks.rados.rados.0.vpm167.stdout:3206: done (3 left) 2014-08-08T10:41:18.564 INFO:tasks.rados.rados.0.vpm167.stdout:3207: delete oid 28 current snap is 422 2014-08-08T10:41:19.224 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 23 v 865 (ObjNum 1078 snap 374 seq_num 1078) dirty exists 2014-08-08T10:41:19.414 INFO:tasks.rados.rados.0.vpm167.stdout:3193: done (3 left) 2014-08-08T10:41:19.415 INFO:tasks.rados.rados.0.vpm167.stdout:3207: done (2 left) 2014-08-08T10:41:19.415 INFO:tasks.rados.rados.0.vpm167.stdout:3208: delete oid 43 current snap is 422 2014-08-08T10:41:19.416 INFO:tasks.rados.rados.0.vpm167.stdout:3208: done (2 left) 2014-08-08T10:41:19.416 INFO:tasks.rados.rados.0.vpm167.stdout:3209: snap_remove snap 417 2014-08-08T10:41:20.085 INFO:tasks.rados.rados.0.vpm167.stdout:3209: done (2 left) 2014-08-08T10:41:20.085 INFO:tasks.rados.rados.0.vpm167.stdout:3210: rollback oid 3 current snap is 422 2014-08-08T10:41:20.085 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 379 2014-08-08T10:41:20.085 INFO:tasks.rados.rados.0.vpm167.stdout:3211: delete oid 29 current snap is 422 2014-08-08T10:41:20.521 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 0 (ObjNum 30933776 snap 0 seq_num 32) dirty dne 2014-08-08T10:41:21.085 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:21.085 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:21.085 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2014-08-08T10:41:21.085 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:41:21.086 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:41:21.086 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-08-08T10:41:21.356 INFO:tasks.rados.rados.0.vpm167.stdout:3199: finishing write tid 1 to vpm1675861-15 2014-08-08T10:41:21.356 INFO:tasks.rados.rados.0.vpm167.stdout:3199: finishing write tid 3 to vpm1675861-15 2014-08-08T10:41:21.357 INFO:tasks.rados.rados.0.vpm167.stdout:3210: done (3 left) 2014-08-08T10:41:21.357 INFO:tasks.rados.rados.0.vpm167.stdout:3211: done (2 left) 2014-08-08T10:41:21.362 INFO:tasks.rados.rados.0.vpm167.stdout:3212: write oid 42 current snap is 422 2014-08-08T10:41:21.362 INFO:tasks.rados.rados.0.vpm167.stdout:3212: seq_num 1208 ranges {495318=672350,1598213=767672,2790225=484292,3295317=1} 2014-08-08T10:41:21.385 INFO:tasks.rados.rados.0.vpm167.stdout:3212: writing vpm1675861-42 from 495318 to 1167668 tid 1 2014-08-08T10:41:21.437 INFO:tasks.rados.rados.0.vpm167.stdout:3212: writing vpm1675861-42 from 1598213 to 2365885 tid 2 2014-08-08T10:41:21.463 INFO:tasks.rados.rados.0.vpm167.stdout:3212: writing vpm1675861-42 from 2790225 to 3274517 tid 3 2014-08-08T10:41:21.465 INFO:tasks.rados.rados.0.vpm167.stdout:3212: writing vpm1675861-42 from 3295317 to 3295318 tid 4 2014-08-08T10:41:21.466 INFO:tasks.rados.rados.0.vpm167.stdout:3213: read oid 34 snap -1 2014-08-08T10:41:21.467 INFO:tasks.rados.rados.0.vpm167.stdout:3214: snap_remove snap 387 2014-08-08T10:41:21.504 INFO:tasks.rados.rados.0.vpm167.stdout:3199: finishing write tid 4 to vpm1675861-15 2014-08-08T10:41:21.505 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 15 v 1673 (ObjNum 1203 snap 421 seq_num 1203) dirty exists 2014-08-08T10:41:21.639 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 1 to vpm1675861-42 2014-08-08T10:41:21.681 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 2 to vpm1675861-42 2014-08-08T10:41:21.690 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 3 to vpm1675861-42 2014-08-08T10:41:21.690 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 4 to vpm1675861-42 2014-08-08T10:41:21.690 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 6 to vpm1675861-42 2014-08-08T10:41:21.713 INFO:tasks.rados.rados.0.vpm167.stdout:3212: finishing write tid 7 to vpm1675861-42 2014-08-08T10:41:21.714 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 2025 (ObjNum 1208 snap 422 seq_num 1208) dirty exists 2014-08-08T10:41:24.908 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1403 (ObjNum 31925616 snap 0 seq_num 48278336) dirty dne 2014-08-08T10:41:24.982 INFO:tasks.rados.rados.0.vpm167.stdout:3186: done (4 left) 2014-08-08T10:41:24.982 INFO:tasks.rados.rados.0.vpm167.stdout:3199: done (3 left) 2014-08-08T10:41:24.982 INFO:tasks.rados.rados.0.vpm167.stdout:3212: done (2 left) 2014-08-08T10:41:24.982 INFO:tasks.rados.rados.0.vpm167.stdout:3213: done (1 left) 2014-08-08T10:41:24.983 INFO:tasks.rados.rados.0.vpm167.stdout:3214: done (0 left) 2014-08-08T10:41:24.983 INFO:tasks.rados.rados.0.vpm167.stdout:3215: delete oid 48 current snap is 422 2014-08-08T10:41:26.195 INFO:tasks.rados.rados.0.vpm167.stdout:3215: done (0 left) 2014-08-08T10:41:26.196 INFO:tasks.rados.rados.0.vpm167.stdout:3216: rollback oid 44 current snap is 422 2014-08-08T10:41:26.196 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 44 to 397 2014-08-08T10:41:26.196 INFO:tasks.rados.rados.0.vpm167.stdout:3217: read oid 12 snap -1 2014-08-08T10:41:26.196 INFO:tasks.rados.rados.0.vpm167.stdout:3218: write oid 48 current snap is 422 2014-08-08T10:41:26.197 INFO:tasks.rados.rados.0.vpm167.stdout:3218: seq_num 1210 ranges {403154=768406,1659181=659486,2911075=292079} 2014-08-08T10:41:26.214 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:26.215 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:26.215 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-08T10:41:26.215 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:41:26.230 INFO:tasks.rados.rados.0.vpm167.stdout:3218: writing vpm1675861-48 from 403154 to 1171560 tid 1 2014-08-08T10:41:26.274 INFO:tasks.rados.rados.0.vpm167.stdout:3218: writing vpm1675861-48 from 1659181 to 2318667 tid 2 2014-08-08T10:41:26.299 INFO:tasks.rados.rados.0.vpm167.stdout:3218: writing vpm1675861-48 from 2911075 to 3203154 tid 3 2014-08-08T10:41:26.300 INFO:tasks.rados.rados.0.vpm167.stdout:3217: expect (ObjNum 950 snap 312 seq_num 950) 2014-08-08T10:41:26.605 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:41:26.700 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T10:41:26.701 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 54' 2014-08-08T10:41:26.947 INFO:tasks.rados.rados.0.vpm167.stdout:3217: done (2 left) 2014-08-08T10:41:26.947 INFO:tasks.rados.rados.0.vpm167.stdout:3219: snap_remove snap 409 2014-08-08T10:41:26.948 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1419 (ObjNum 1022 snap 348 seq_num 1022) dirty exists 2014-08-08T10:41:26.948 INFO:tasks.rados.rados.0.vpm167.stdout:3218: finishing write tid 1 to vpm1675861-48 2014-08-08T10:41:26.948 INFO:tasks.rados.rados.0.vpm167.stdout:3218: finishing write tid 2 to vpm1675861-48 2014-08-08T10:41:26.948 INFO:tasks.rados.rados.0.vpm167.stdout:3218: finishing write tid 3 to vpm1675861-48 2014-08-08T10:41:26.949 INFO:tasks.rados.rados.0.vpm167.stdout:3218: finishing write tid 5 to vpm1675861-48 2014-08-08T10:41:26.949 INFO:tasks.rados.rados.0.vpm167.stdout:3218: finishing write tid 6 to vpm1675861-48 2014-08-08T10:41:26.949 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 48 v 1681 (ObjNum 1210 snap 422 seq_num 1210) dirty exists 2014-08-08T10:41:27.570 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 2 pg_num to 54 2014-08-08T10:41:27.669 INFO:tasks.rados.rados.0.vpm167.stdout:3216: done (2 left) 2014-08-08T10:41:27.669 INFO:tasks.rados.rados.0.vpm167.stdout:3218: done (1 left) 2014-08-08T10:41:27.669 INFO:tasks.rados.rados.0.vpm167.stdout:3219: done (0 left) 2014-08-08T10:41:27.669 INFO:tasks.rados.rados.0.vpm167.stdout:3220: snap_remove snap 369 2014-08-08T10:41:28.714 INFO:tasks.rados.rados.0.vpm167.stdout:3220: done (0 left) 2014-08-08T10:41:28.714 INFO:tasks.rados.rados.0.vpm167.stdout:3221: snap_create 2014-08-08T10:41:29.782 INFO:tasks.rados.rados.0.vpm167.stdout:3221: done (0 left) 2014-08-08T10:41:29.782 INFO:tasks.rados.rados.0.vpm167.stdout:3222: write oid 41 current snap is 423 2014-08-08T10:41:29.782 INFO:tasks.rados.rados.0.vpm167.stdout:3222: seq_num 1211 ranges {596559=400000} 2014-08-08T10:41:29.796 INFO:tasks.rados.rados.0.vpm167.stdout:3222: writing vpm1675861-41 from 596559 to 996559 tid 1 2014-08-08T10:41:29.799 INFO:tasks.rados.rados.0.vpm167.stdout:3223: write oid 32 current snap is 423 2014-08-08T10:41:29.800 INFO:tasks.rados.rados.0.vpm167.stdout:3223: seq_num 1212 ranges {692812=1} 2014-08-08T10:41:29.805 INFO:tasks.rados.rados.0.vpm167.stdout:3223: writing vpm1675861-32 from 692812 to 692813 tid 1 2014-08-08T10:41:29.807 INFO:tasks.rados.rados.0.vpm167.stdout:3224: write oid 15 current snap is 423 2014-08-08T10:41:29.807 INFO:tasks.rados.rados.0.vpm167.stdout:3224: seq_num 1213 ranges {504395=400000} 2014-08-08T10:41:29.833 INFO:tasks.rados.rados.0.vpm167.stdout:3224: writing vpm1675861-15 from 504395 to 904395 tid 1 2014-08-08T10:41:29.838 INFO:tasks.rados.rados.0.vpm167.stdout:3225: read oid 38 snap -1 2014-08-08T10:41:29.839 INFO:tasks.rados.rados.0.vpm167.stdout:3226: delete oid 20 current snap is 423 2014-08-08T10:41:29.848 INFO:tasks.rados.rados.0.vpm167.stdout:3222: finishing write tid 1 to vpm1675861-41 2014-08-08T10:41:29.851 INFO:tasks.rados.rados.0.vpm167.stdout:3222: finishing write tid 3 to vpm1675861-41 2014-08-08T10:41:29.863 INFO:tasks.rados.rados.0.vpm167.stdout:3222: finishing write tid 4 to vpm1675861-41 2014-08-08T10:41:29.863 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1722 (ObjNum 1211 snap 423 seq_num 1211) dirty exists 2014-08-08T10:41:29.881 INFO:tasks.rados.rados.0.vpm167.stdout:3222: done (4 left) 2014-08-08T10:41:29.882 INFO:tasks.rados.rados.0.vpm167.stdout:3226: done (3 left) 2014-08-08T10:41:29.883 INFO:tasks.rados.rados.0.vpm167.stdout:3227: snap_create 2014-08-08T10:41:29.899 INFO:tasks.rados.rados.0.vpm167.stdout:3225: expect (ObjNum 1157 snap 407 seq_num 1157) 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3224: finishing write tid 1 to vpm1675861-15 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3224: finishing write tid 3 to vpm1675861-15 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3223: finishing write tid 1 to vpm1675861-32 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3223: finishing write tid 3 to vpm1675861-32 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3223: finishing write tid 4 to vpm1675861-32 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 32 v 2140 (ObjNum 1212 snap 423 seq_num 1212) dirty exists 2014-08-08T10:41:30.409 INFO:tasks.rados.rados.0.vpm167.stdout:3224: finishing write tid 4 to vpm1675861-15 2014-08-08T10:41:30.410 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 15 v 1702 (ObjNum 1213 snap 423 seq_num 1213) dirty exists 2014-08-08T10:41:30.899 INFO:tasks.rados.rados.0.vpm167.stdout:3223: done (3 left) 2014-08-08T10:41:30.900 INFO:tasks.rados.rados.0.vpm167.stdout:3224: done (2 left) 2014-08-08T10:41:30.900 INFO:tasks.rados.rados.0.vpm167.stdout:3225: done (1 left) 2014-08-08T10:41:30.901 INFO:tasks.rados.rados.0.vpm167.stdout:3227: done (0 left) 2014-08-08T10:41:30.901 INFO:tasks.rados.rados.0.vpm167.stdout:3228: snap_create 2014-08-08T10:41:31.985 INFO:tasks.rados.rados.0.vpm167.stdout:3228: done (0 left) 2014-08-08T10:41:31.985 INFO:tasks.rados.rados.0.vpm167.stdout:3229: write oid 34 current snap is 425 2014-08-08T10:41:31.985 INFO:tasks.rados.rados.0.vpm167.stdout:3229: seq_num 1215 ranges {793030=698596,2138450=254580} 2014-08-08T10:41:32.028 INFO:tasks.rados.rados.0.vpm167.stdout:3229: writing vpm1675861-34 from 793030 to 1491626 tid 1 2014-08-08T10:41:32.052 INFO:tasks.rados.rados.0.vpm167.stdout:3229: writing vpm1675861-34 from 2138450 to 2393030 tid 2 2014-08-08T10:41:32.054 INFO:tasks.rados.rados.0.vpm167.stdout:3230: write oid 13 current snap is 425 2014-08-08T10:41:32.054 INFO:tasks.rados.rados.0.vpm167.stdout:3230: seq_num 1216 ranges {488260=670400,1851558=236702} 2014-08-08T10:41:32.092 INFO:tasks.rados.rados.0.vpm167.stdout:3230: writing vpm1675861-13 from 488260 to 1158660 tid 1 2014-08-08T10:41:32.114 INFO:tasks.rados.rados.0.vpm167.stdout:3230: writing vpm1675861-13 from 1851558 to 2088260 tid 2 2014-08-08T10:41:32.116 INFO:tasks.rados.rados.0.vpm167.stdout:3231: write oid 36 current snap is 425 2014-08-08T10:41:32.116 INFO:tasks.rados.rados.0.vpm167.stdout:3231: seq_num 1217 ranges {681666=643229,1912443=687748,3328805=552861} 2014-08-08T10:41:32.161 INFO:tasks.rados.rados.0.vpm167.stdout:3231: writing vpm1675861-36 from 681666 to 1324895 tid 1 2014-08-08T10:41:32.212 INFO:tasks.rados.rados.0.vpm167.stdout:3231: writing vpm1675861-36 from 1912443 to 2600191 tid 2 2014-08-08T10:41:32.250 INFO:tasks.rados.rados.0.vpm167.stdout:3231: writing vpm1675861-36 from 3328805 to 3881666 tid 3 2014-08-08T10:41:32.252 INFO:tasks.rados.rados.0.vpm167.stdout:3229: finishing write tid 1 to vpm1675861-34 2014-08-08T10:41:32.253 INFO:tasks.rados.rados.0.vpm167.stdout:3229: finishing write tid 2 to vpm1675861-34 2014-08-08T10:41:32.253 INFO:tasks.rados.rados.0.vpm167.stdout:3229: finishing write tid 4 to vpm1675861-34 2014-08-08T10:41:32.255 INFO:tasks.rados.rados.0.vpm167.stdout:3229: finishing write tid 5 to vpm1675861-34 2014-08-08T10:41:32.255 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1725 (ObjNum 1215 snap 425 seq_num 1215) dirty exists 2014-08-08T10:41:32.255 INFO:tasks.rados.rados.0.vpm167.stdout:3230: finishing write tid 1 to vpm1675861-13 2014-08-08T10:41:32.255 INFO:tasks.rados.rados.0.vpm167.stdout:3230: finishing write tid 2 to vpm1675861-13 2014-08-08T10:41:32.256 INFO:tasks.rados.rados.0.vpm167.stdout:3230: finishing write tid 4 to vpm1675861-13 2014-08-08T10:41:32.256 INFO:tasks.rados.rados.0.vpm167.stdout:3230: finishing write tid 5 to vpm1675861-13 2014-08-08T10:41:32.256 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1706 (ObjNum 1216 snap 425 seq_num 1216) dirty exists 2014-08-08T10:41:32.256 INFO:tasks.rados.rados.0.vpm167.stdout:3229: done (2 left) 2014-08-08T10:41:32.257 INFO:tasks.rados.rados.0.vpm167.stdout:3230: done (1 left) 2014-08-08T10:41:32.257 INFO:tasks.rados.rados.0.vpm167.stdout:3232: read oid 43 snap -1 2014-08-08T10:41:32.257 INFO:tasks.rados.rados.0.vpm167.stdout:3233: snap_create 2014-08-08T10:41:32.286 INFO:tasks.rados.rados.0.vpm167.stdout:3231: finishing write tid 1 to vpm1675861-36 2014-08-08T10:41:32.287 INFO:tasks.rados.rados.0.vpm167.stdout:3231: finishing write tid 2 to vpm1675861-36 2014-08-08T10:41:32.310 INFO:tasks.rados.rados.0.vpm167.stdout:3231: finishing write tid 3 to vpm1675861-36 2014-08-08T10:41:32.312 INFO:tasks.rados.rados.0.vpm167.stdout:3231: finishing write tid 5 to vpm1675861-36 2014-08-08T10:41:32.317 INFO:tasks.rados.rados.0.vpm167.stdout:3231: finishing write tid 6 to vpm1675861-36 2014-08-08T10:41:32.317 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 36 v 2044 (ObjNum 1217 snap 425 seq_num 1217) dirty exists 2014-08-08T10:41:32.623 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:32.624 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:32.624 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2014-08-08T10:41:32.624 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:41:32.625 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:41:32.625 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-08-08T10:41:33.042 INFO:tasks.rados.rados.0.vpm167.stdout:3231: done (2 left) 2014-08-08T10:41:33.042 INFO:tasks.rados.rados.0.vpm167.stdout:3232: done (1 left) 2014-08-08T10:41:33.043 INFO:tasks.rados.rados.0.vpm167.stdout:3233: done (0 left) 2014-08-08T10:41:33.043 INFO:tasks.rados.rados.0.vpm167.stdout:3234: snap_create 2014-08-08T10:41:34.219 INFO:tasks.rados.rados.0.vpm167.stdout:3234: done (0 left) 2014-08-08T10:41:34.219 INFO:tasks.rados.rados.0.vpm167.stdout:3235: write oid 39 current snap is 427 2014-08-08T10:41:34.219 INFO:tasks.rados.rados.0.vpm167.stdout:3235: seq_num 1218 ranges {93246=1} 2014-08-08T10:41:34.220 INFO:tasks.rados.rados.0.vpm167.stdout:3235: writing vpm1675861-39 from 93246 to 93247 tid 1 2014-08-08T10:41:34.221 INFO:tasks.rados.rados.0.vpm167.stdout:3236: write oid 33 current snap is 427 2014-08-08T10:41:34.221 INFO:tasks.rados.rados.0.vpm167.stdout:3236: seq_num 1219 ranges {589501=621910,2007459=428139,3199032=590469} 2014-08-08T10:41:34.245 INFO:tasks.rados.rados.0.vpm167.stdout:3236: writing vpm1675861-33 from 589501 to 1211411 tid 1 2014-08-08T10:41:34.273 INFO:tasks.rados.rados.0.vpm167.stdout:3236: writing vpm1675861-33 from 2007459 to 2435598 tid 2 2014-08-08T10:41:34.298 INFO:tasks.rados.rados.0.vpm167.stdout:3236: writing vpm1675861-33 from 3199032 to 3789501 tid 3 2014-08-08T10:41:34.300 INFO:tasks.rados.rados.0.vpm167.stdout:3237: read oid 10 snap -1 2014-08-08T10:41:34.301 INFO:tasks.rados.rados.0.vpm167.stdout:3238: write oid 3 current snap is 427 2014-08-08T10:41:34.302 INFO:tasks.rados.rados.0.vpm167.stdout:3238: seq_num 1220 ranges {533306=1} 2014-08-08T10:41:34.306 INFO:tasks.rados.rados.0.vpm167.stdout:3238: writing vpm1675861-3 from 533306 to 533307 tid 1 2014-08-08T10:41:34.307 INFO:tasks.rados.rados.0.vpm167.stdout:3239: write oid 4 current snap is 427 2014-08-08T10:41:34.308 INFO:tasks.rados.rados.0.vpm167.stdout:3239: seq_num 1221 ranges {478137=717966,1278136=1} 2014-08-08T10:41:34.333 INFO:tasks.rados.rados.0.vpm167.stdout:3239: writing vpm1675861-4 from 478137 to 1196103 tid 1 2014-08-08T10:41:34.342 INFO:tasks.rados.rados.0.vpm167.stdout:3239: writing vpm1675861-4 from 1278136 to 1278137 tid 2 2014-08-08T10:41:34.343 INFO:tasks.rados.rados.0.vpm167.stdout:3240: read oid 49 snap -1 2014-08-08T10:41:34.343 INFO:tasks.rados.rados.0.vpm167.stdout:3241: snap_remove snap 389 2014-08-08T10:41:34.376 INFO:tasks.rados.rados.0.vpm167.stdout:3237: expect (ObjNum 1191 snap 415 seq_num 1191) 2014-08-08T10:41:36.356 INFO:tasks.rados.rados.0.vpm167.stdout:3237: done (6 left) 2014-08-08T10:41:36.357 INFO:tasks.rados.rados.0.vpm167.stdout:3240: done (5 left) 2014-08-08T10:41:36.357 INFO:tasks.rados.rados.0.vpm167.stdout:3241: done (4 left) 2014-08-08T10:41:36.357 INFO:tasks.rados.rados.0.vpm167.stdout:3242: rollback oid 50 current snap is 427 2014-08-08T10:41:36.358 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 50 to 415 2014-08-08T10:41:36.358 INFO:tasks.rados.rados.0.vpm167.stdout:3243: delete oid 9 current snap is 427 2014-08-08T10:41:36.358 INFO:tasks.rados.rados.0.vpm167.stdout:3236: finishing write tid 1 to vpm1675861-33 2014-08-08T10:41:36.359 INFO:tasks.rados.rados.0.vpm167.stdout:3235: finishing write tid 1 to vpm1675861-39 2014-08-08T10:41:36.359 INFO:tasks.rados.rados.0.vpm167.stdout:3235: finishing write tid 3 to vpm1675861-39 2014-08-08T10:41:36.359 INFO:tasks.rados.rados.0.vpm167.stdout:3235: finishing write tid 4 to vpm1675861-39 2014-08-08T10:41:36.359 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 39 v 1426 (ObjNum 1218 snap 427 seq_num 1218) dirty exists 2014-08-08T10:41:36.360 INFO:tasks.rados.rados.0.vpm167.stdout:3238: finishing write tid 1 to vpm1675861-3 2014-08-08T10:41:36.360 INFO:tasks.rados.rados.0.vpm167.stdout:3238: finishing write tid 3 to vpm1675861-3 2014-08-08T10:41:36.361 INFO:tasks.rados.rados.0.vpm167.stdout:3236: finishing write tid 2 to vpm1675861-33 2014-08-08T10:41:36.361 INFO:tasks.rados.rados.0.vpm167.stdout:3239: finishing write tid 1 to vpm1675861-4 2014-08-08T10:41:36.361 INFO:tasks.rados.rados.0.vpm167.stdout:3236: finishing write tid 3 to vpm1675861-33 2014-08-08T10:41:36.361 INFO:tasks.rados.rados.0.vpm167.stdout:3239: finishing write tid 2 to vpm1675861-4 2014-08-08T10:41:36.362 INFO:tasks.rados.rados.0.vpm167.stdout:3236: finishing write tid 5 to vpm1675861-33 2014-08-08T10:41:36.362 INFO:tasks.rados.rados.0.vpm167.stdout:3239: finishing write tid 4 to vpm1675861-4 2014-08-08T10:41:36.362 INFO:tasks.rados.rados.0.vpm167.stdout:3238: finishing write tid 4 to vpm1675861-3 2014-08-08T10:41:36.363 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 803 (ObjNum 1220 snap 427 seq_num 1220) dirty exists 2014-08-08T10:41:36.363 INFO:tasks.rados.rados.0.vpm167.stdout:3236: finishing write tid 6 to vpm1675861-33 2014-08-08T10:41:36.363 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1711 (ObjNum 1219 snap 427 seq_num 1219) dirty exists 2014-08-08T10:41:36.363 INFO:tasks.rados.rados.0.vpm167.stdout:3239: finishing write tid 5 to vpm1675861-4 2014-08-08T10:41:36.363 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 4 v 1160 (ObjNum 1221 snap 427 seq_num 1221) dirty exists 2014-08-08T10:41:36.468 INFO:tasks.rados.rados.0.vpm167.stdout:3235: done (5 left) 2014-08-08T10:41:36.468 INFO:tasks.rados.rados.0.vpm167.stdout:3236: done (4 left) 2014-08-08T10:41:36.468 INFO:tasks.rados.rados.0.vpm167.stdout:3238: done (3 left) 2014-08-08T10:41:36.469 INFO:tasks.rados.rados.0.vpm167.stdout:3239: done (2 left) 2014-08-08T10:41:36.469 INFO:tasks.rados.rados.0.vpm167.stdout:3243: done (1 left) 2014-08-08T10:41:36.470 INFO:tasks.rados.rados.0.vpm167.stdout:3244: write oid 29 current snap is 427 2014-08-08T10:41:36.470 INFO:tasks.rados.rados.0.vpm167.stdout:3244: seq_num 1223 ranges {634548=697671,1992819=241729} 2014-08-08T10:41:36.496 INFO:tasks.rados.rados.0.vpm167.stdout:3244: writing vpm1675861-29 from 634548 to 1332219 tid 1 2014-08-08T10:41:36.512 INFO:tasks.rados.rados.0.vpm167.stdout:3244: writing vpm1675861-29 from 1992819 to 2234548 tid 2 2014-08-08T10:41:36.513 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 50 v 750 (ObjNum 4000000 snap 0 seq_num 400000) dirty dne 2014-08-08T10:41:36.514 INFO:tasks.rados.rados.0.vpm167.stdout:3242: done (1 left) 2014-08-08T10:41:36.514 INFO:tasks.rados.rados.0.vpm167.stdout:3245: write oid 26 current snap is 427 2014-08-08T10:41:36.514 INFO:tasks.rados.rados.0.vpm167.stdout:3245: seq_num 1224 ranges {578354=420899,1554503=681963,2701287=277067} 2014-08-08T10:41:36.533 INFO:tasks.rados.rados.0.vpm167.stdout:3245: writing vpm1675861-26 from 578354 to 999253 tid 1 2014-08-08T10:41:36.560 INFO:tasks.rados.rados.0.vpm167.stdout:3245: writing vpm1675861-26 from 1554503 to 2236466 tid 2 2014-08-08T10:41:36.576 INFO:tasks.rados.rados.0.vpm167.stdout:3245: writing vpm1675861-26 from 2701287 to 2978354 tid 3 2014-08-08T10:41:36.579 INFO:tasks.rados.rados.0.vpm167.stdout:3246: read oid 13 snap -1 2014-08-08T10:41:36.579 INFO:tasks.rados.rados.0.vpm167.stdout:3247: read oid 39 snap -1 2014-08-08T10:41:36.580 INFO:tasks.rados.rados.0.vpm167.stdout:3248: delete oid 35 current snap is 427 2014-08-08T10:41:36.604 INFO:tasks.rados.rados.0.vpm167.stdout:3246: expect (ObjNum 1216 snap 425 seq_num 1216) 2014-08-08T10:41:37.305 INFO:tasks.rados.rados.0.vpm167.stdout:3246: done (4 left) 2014-08-08T10:41:37.306 INFO:tasks.rados.rados.0.vpm167.stdout:3248: done (3 left) 2014-08-08T10:41:37.306 INFO:tasks.rados.rados.0.vpm167.stdout:3249: rollback oid 46 current snap is 427 2014-08-08T10:41:37.306 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 46 to 396 2014-08-08T10:41:37.307 INFO:tasks.rados.rados.0.vpm167.stdout:3250: write oid 6 current snap is 427 2014-08-08T10:41:37.307 INFO:tasks.rados.rados.0.vpm167.stdout:3250: seq_num 1226 ranges {618413=516956,1899119=706001,3238009=180404} 2014-08-08T10:41:37.327 INFO:tasks.rados.rados.0.vpm167.stdout:3250: writing vpm1675861-6 from 618413 to 1135369 tid 1 2014-08-08T10:41:37.368 INFO:tasks.rados.rados.0.vpm167.stdout:3250: writing vpm1675861-6 from 1899119 to 2605120 tid 2 2014-08-08T10:41:37.381 INFO:tasks.rados.rados.0.vpm167.stdout:3250: writing vpm1675861-6 from 3238009 to 3418413 tid 3 2014-08-08T10:41:37.382 INFO:tasks.rados.rados.0.vpm167.stdout:3247: expect (ObjNum 1218 snap 427 seq_num 1218) 2014-08-08T10:41:37.390 INFO:tasks.rados.rados.0.vpm167.stdout:3247: done (4 left) 2014-08-08T10:41:37.391 INFO:tasks.rados.rados.0.vpm167.stdout:3251: snap_create 2014-08-08T10:41:37.392 INFO:tasks.rados.rados.0.vpm167.stdout:3244: finishing write tid 1 to vpm1675861-29 2014-08-08T10:41:37.392 INFO:tasks.rados.rados.0.vpm167.stdout:3244: finishing write tid 2 to vpm1675861-29 2014-08-08T10:41:37.392 INFO:tasks.rados.rados.0.vpm167.stdout:3244: finishing write tid 4 to vpm1675861-29 2014-08-08T10:41:37.393 INFO:tasks.rados.rados.0.vpm167.stdout:3245: finishing write tid 1 to vpm1675861-26 2014-08-08T10:41:37.394 INFO:tasks.rados.rados.0.vpm167.stdout:3245: finishing write tid 2 to vpm1675861-26 2014-08-08T10:41:37.394 INFO:tasks.rados.rados.0.vpm167.stdout:3245: finishing write tid 3 to vpm1675861-26 2014-08-08T10:41:37.394 INFO:tasks.rados.rados.0.vpm167.stdout:3245: finishing write tid 5 to vpm1675861-26 2014-08-08T10:41:37.394 INFO:tasks.rados.rados.0.vpm167.stdout:3245: finishing write tid 6 to vpm1675861-26 2014-08-08T10:41:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 892 (ObjNum 1224 snap 427 seq_num 1224) dirty exists 2014-08-08T10:41:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:3244: finishing write tid 5 to vpm1675861-29 2014-08-08T10:41:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 1095 (ObjNum 1223 snap 427 seq_num 1223) dirty exists 2014-08-08T10:41:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 46 v 1721 (ObjNum 1060 snap 364 seq_num 1060) dirty exists 2014-08-08T10:41:37.411 INFO:tasks.rados.rados.0.vpm167.stdout:3250: finishing write tid 1 to vpm1675861-6 2014-08-08T10:41:37.458 INFO:tasks.rados.rados.0.vpm167.stdout:3250: finishing write tid 2 to vpm1675861-6 2014-08-08T10:41:37.462 INFO:tasks.rados.rados.0.vpm167.stdout:3250: finishing write tid 3 to vpm1675861-6 2014-08-08T10:41:37.465 INFO:tasks.rados.rados.0.vpm167.stdout:3250: finishing write tid 5 to vpm1675861-6 2014-08-08T10:41:37.467 INFO:tasks.rados.rados.0.vpm167.stdout:3250: finishing write tid 6 to vpm1675861-6 2014-08-08T10:41:37.468 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 6 v 1737 (ObjNum 1226 snap 427 seq_num 1226) dirty exists 2014-08-08T10:41:37.718 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:37.718 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:37.719 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2014-08-08T10:41:37.719 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:41:37.719 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:41:37.719 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-08-08T10:41:37.824 INFO:tasks.rados.rados.0.vpm167.stdout:3244: done (4 left) 2014-08-08T10:41:37.825 INFO:tasks.rados.rados.0.vpm167.stdout:3245: done (3 left) 2014-08-08T10:41:37.825 INFO:tasks.rados.rados.0.vpm167.stdout:3249: done (2 left) 2014-08-08T10:41:37.825 INFO:tasks.rados.rados.0.vpm167.stdout:3250: done (1 left) 2014-08-08T10:41:37.825 INFO:tasks.rados.rados.0.vpm167.stdout:3251: done (0 left) 2014-08-08T10:41:37.826 INFO:tasks.rados.rados.0.vpm167.stdout:3252: write oid 10 current snap is 428 2014-08-08T10:41:37.826 INFO:tasks.rados.rados.0.vpm167.stdout:3252: seq_num 1227 ranges {679594=1} 2014-08-08T10:41:37.833 INFO:tasks.rados.rados.0.vpm167.stdout:3252: writing vpm1675861-10 from 679594 to 679595 tid 1 2014-08-08T10:41:37.834 INFO:tasks.rados.rados.0.vpm167.stdout:3253: read oid 2 snap -1 2014-08-08T10:41:37.835 INFO:tasks.rados.rados.0.vpm167.stdout:3254: write oid 20 current snap is 428 2014-08-08T10:41:37.835 INFO:tasks.rados.rados.0.vpm167.stdout:3254: seq_num 1228 ranges {623401=745237,1423400=1} 2014-08-08T10:41:37.868 INFO:tasks.rados.rados.0.vpm167.stdout:3254: writing vpm1675861-20 from 623401 to 1368638 tid 1 2014-08-08T10:41:37.871 INFO:tasks.rados.rados.0.vpm167.stdout:3254: writing vpm1675861-20 from 1423400 to 1423401 tid 2 2014-08-08T10:41:37.872 INFO:tasks.rados.rados.0.vpm167.stdout:3255: read oid 31 snap -1 2014-08-08T10:41:37.872 INFO:tasks.rados.rados.0.vpm167.stdout:3256: rollback oid 45 current snap is 428 2014-08-08T10:41:37.872 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 45 to 423 2014-08-08T10:41:37.873 INFO:tasks.rados.rados.0.vpm167.stdout:3257: read oid 5 snap -1 2014-08-08T10:41:37.874 INFO:tasks.rados.rados.0.vpm167.stdout:3258: snap_create 2014-08-08T10:41:37.930 INFO:tasks.rados.rados.0.vpm167.stdout:3255: expect (ObjNum 1173 snap 413 seq_num 1173) 2014-08-08T10:41:39.072 INFO:tasks.rados.rados.0.vpm167.stdout:3255: done (6 left) 2014-08-08T10:41:39.073 INFO:tasks.rados.rados.0.vpm167.stdout:3258: done (5 left) 2014-08-08T10:41:39.073 INFO:tasks.rados.rados.0.vpm167.stdout:3259: write oid 34 current snap is 429 2014-08-08T10:41:39.073 INFO:tasks.rados.rados.0.vpm167.stdout:3259: seq_num 1229 ranges {719655=400000} 2014-08-08T10:41:39.090 INFO:tasks.rados.rados.0.vpm167.stdout:3259: writing vpm1675861-34 from 719655 to 1119655 tid 1 2014-08-08T10:41:39.091 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 0 (ObjNum 4000000 snap 0 seq_num 400000) dirty dne 2014-08-08T10:41:39.091 INFO:tasks.rados.rados.0.vpm167.stdout:3257: expect (ObjNum 1079 snap 374 seq_num 1079) 2014-08-08T10:41:39.093 INFO:tasks.rados.rados.0.vpm167.stdout:3252: finishing write tid 1 to vpm1675861-10 2014-08-08T10:41:39.093 INFO:tasks.rados.rados.0.vpm167.stdout:3253: expect (ObjNum 1101 snap 385 seq_num 1101) 2014-08-08T10:41:40.139 INFO:tasks.rados.rados.0.vpm167.stdout:3253: done (5 left) 2014-08-08T10:41:40.141 INFO:tasks.rados.rados.0.vpm167.stdout:3256: done (4 left) 2014-08-08T10:41:40.141 INFO:tasks.rados.rados.0.vpm167.stdout:3257: done (3 left) 2014-08-08T10:41:40.141 INFO:tasks.rados.rados.0.vpm167.stdout:3260: delete oid 42 current snap is 429 2014-08-08T10:41:40.142 INFO:tasks.rados.rados.0.vpm167.stdout:3252: finishing write tid 3 to vpm1675861-10 2014-08-08T10:41:40.142 INFO:tasks.rados.rados.0.vpm167.stdout:3254: finishing write tid 1 to vpm1675861-20 2014-08-08T10:41:40.142 INFO:tasks.rados.rados.0.vpm167.stdout:3254: finishing write tid 2 to vpm1675861-20 2014-08-08T10:41:40.142 INFO:tasks.rados.rados.0.vpm167.stdout:3254: finishing write tid 4 to vpm1675861-20 2014-08-08T10:41:40.142 INFO:tasks.rados.rados.0.vpm167.stdout:3252: finishing write tid 4 to vpm1675861-10 2014-08-08T10:41:40.143 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 10 v 923 (ObjNum 1227 snap 428 seq_num 1227) dirty exists 2014-08-08T10:41:40.143 INFO:tasks.rados.rados.0.vpm167.stdout:3254: finishing write tid 5 to vpm1675861-20 2014-08-08T10:41:40.144 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 20 v 927 (ObjNum 1228 snap 428 seq_num 1228) dirty exists 2014-08-08T10:41:40.144 INFO:tasks.rados.rados.0.vpm167.stdout:3259: finishing write tid 1 to vpm1675861-34 2014-08-08T10:41:40.144 INFO:tasks.rados.rados.0.vpm167.stdout:3259: finishing write tid 3 to vpm1675861-34 2014-08-08T10:41:40.144 INFO:tasks.rados.rados.0.vpm167.stdout:3259: finishing write tid 4 to vpm1675861-34 2014-08-08T10:41:40.144 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1740 (ObjNum 1229 snap 429 seq_num 1229) dirty exists 2014-08-08T10:41:40.149 INFO:tasks.rados.rados.0.vpm167.stdout:3252: done (3 left) 2014-08-08T10:41:40.150 INFO:tasks.rados.rados.0.vpm167.stdout:3254: done (2 left) 2014-08-08T10:41:40.150 INFO:tasks.rados.rados.0.vpm167.stdout:3259: done (1 left) 2014-08-08T10:41:40.151 INFO:tasks.rados.rados.0.vpm167.stdout:3260: done (0 left) 2014-08-08T10:41:40.151 INFO:tasks.rados.rados.0.vpm167.stdout:3261: rollback oid 17 current snap is 429 2014-08-08T10:41:40.151 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 17 to 410 2014-08-08T10:41:40.152 INFO:tasks.rados.rados.0.vpm167.stdout:3262: write oid 3 current snap is 429 2014-08-08T10:41:40.152 INFO:tasks.rados.rados.0.vpm167.stdout:3262: seq_num 1231 ranges {724642=695722,1943317=559825,2953769=170873} 2014-08-08T10:41:40.180 INFO:tasks.rados.rados.0.vpm167.stdout:3262: writing vpm1675861-3 from 724642 to 1420364 tid 1 2014-08-08T10:41:40.214 INFO:tasks.rados.rados.0.vpm167.stdout:3262: writing vpm1675861-3 from 1943317 to 2503142 tid 2 2014-08-08T10:41:40.225 INFO:tasks.rados.rados.0.vpm167.stdout:3262: writing vpm1675861-3 from 2953769 to 3124642 tid 3 2014-08-08T10:41:40.227 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2151 (ObjNum 4000000 snap 0 seq_num 400000) dirty dne 2014-08-08T10:41:40.227 INFO:tasks.rados.rados.0.vpm167.stdout:3262: finishing write tid 1 to vpm1675861-3 2014-08-08T10:41:40.228 INFO:tasks.rados.rados.0.vpm167.stdout:3261: done (1 left) 2014-08-08T10:41:40.228 INFO:tasks.rados.rados.0.vpm167.stdout:3263: rollback oid 25 current snap is 429 2014-08-08T10:41:40.228 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 25 to 370 2014-08-08T10:41:40.229 INFO:tasks.rados.rados.0.vpm167.stdout:3264: snap_remove snap 425 2014-08-08T10:41:40.242 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 25 v 834 (ObjNum 1048 snap 360 seq_num 1048) dirty exists 2014-08-08T10:41:40.246 INFO:tasks.rados.rados.0.vpm167.stdout:3262: finishing write tid 2 to vpm1675861-3 2014-08-08T10:41:40.253 INFO:tasks.rados.rados.0.vpm167.stdout:3262: finishing write tid 3 to vpm1675861-3 2014-08-08T10:41:40.259 INFO:tasks.rados.rados.0.vpm167.stdout:3262: finishing write tid 5 to vpm1675861-3 2014-08-08T10:41:40.260 INFO:tasks.rados.rados.0.vpm167.stdout:3262: finishing write tid 6 to vpm1675861-3 2014-08-08T10:41:40.261 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 810 (ObjNum 1231 snap 429 seq_num 1231) dirty exists 2014-08-08T10:41:41.117 INFO:tasks.rados.rados.0.vpm167.stdout:3262: done (2 left) 2014-08-08T10:41:41.117 INFO:tasks.rados.rados.0.vpm167.stdout:3263: done (1 left) 2014-08-08T10:41:41.117 INFO:tasks.rados.rados.0.vpm167.stdout:3264: done (0 left) 2014-08-08T10:41:41.118 INFO:tasks.rados.rados.0.vpm167.stdout:3265: snap_create 2014-08-08T10:41:42.188 INFO:tasks.rados.rados.0.vpm167.stdout:3265: done (0 left) 2014-08-08T10:41:42.188 INFO:tasks.rados.rados.0.vpm167.stdout:3266: read oid 34 snap -1 2014-08-08T10:41:42.188 INFO:tasks.rados.rados.0.vpm167.stdout:3267: read oid 5 snap 424 2014-08-08T10:41:42.189 INFO:tasks.rados.rados.0.vpm167.stdout:3268: snap_create 2014-08-08T10:41:42.227 INFO:tasks.rados.rados.0.vpm167.stdout:3267: expect (ObjNum 1079 snap 374 seq_num 1079) 2014-08-08T10:41:42.281 INFO:tasks.rados.rados.0.vpm167.stdout:3266: expect (ObjNum 1229 snap 429 seq_num 1229) 2014-08-08T10:41:42.975 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:42.975 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:42.975 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2014-08-08T10:41:42.976 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:41:43.324 INFO:tasks.rados.rados.0.vpm167.stdout:3266: done (2 left) 2014-08-08T10:41:43.325 INFO:tasks.rados.rados.0.vpm167.stdout:3267: done (1 left) 2014-08-08T10:41:43.325 INFO:tasks.rados.rados.0.vpm167.stdout:3268: done (0 left) 2014-08-08T10:41:43.325 INFO:tasks.rados.rados.0.vpm167.stdout:3269: snap_create 2014-08-08T10:41:43.466 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:41:43.527 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T10:41:43.527 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_1 pg_num 56' 2014-08-08T10:41:44.354 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 4 pg_num to 56 2014-08-08T10:41:44.488 INFO:tasks.rados.rados.0.vpm167.stdout:3269: done (0 left) 2014-08-08T10:41:44.488 INFO:tasks.rados.rados.0.vpm167.stdout:3270: rollback oid 50 current snap is 432 2014-08-08T10:41:44.488 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 50 to 344 2014-08-08T10:41:44.488 INFO:tasks.rados.rados.0.vpm167.stdout:3271: read oid 17 snap -1 2014-08-08T10:41:44.488 INFO:tasks.rados.rados.0.vpm167.stdout:3272: snap_remove snap 358 2014-08-08T10:41:44.559 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 50 v 754 (ObjNum 871 snap 293 seq_num 871) dirty exists 2014-08-08T10:41:45.436 INFO:tasks.rados.rados.0.vpm167.stdout:3270: done (2 left) 2014-08-08T10:41:45.436 INFO:tasks.rados.rados.0.vpm167.stdout:3271: done (1 left) 2014-08-08T10:41:45.436 INFO:tasks.rados.rados.0.vpm167.stdout:3272: done (0 left) 2014-08-08T10:41:45.437 INFO:tasks.rados.rados.0.vpm167.stdout:3273: delete oid 36 current snap is 432 2014-08-08T10:41:45.698 INFO:tasks.rados.rados.0.vpm167.stdout:3273: done (0 left) 2014-08-08T10:41:45.699 INFO:tasks.rados.rados.0.vpm167.stdout:3274: write oid 28 current snap is 432 2014-08-08T10:41:45.702 INFO:tasks.rados.rados.0.vpm167.stdout:3274: seq_num 1233 ranges {515102=791778,1886909=450615,2515101=1} 2014-08-08T10:41:45.734 INFO:tasks.rados.rados.0.vpm167.stdout:3274: writing vpm1675861-28 from 515102 to 1306880 tid 1 2014-08-08T10:41:45.759 INFO:tasks.rados.rados.0.vpm167.stdout:3274: writing vpm1675861-28 from 1886909 to 2337524 tid 2 2014-08-08T10:41:45.761 INFO:tasks.rados.rados.0.vpm167.stdout:3274: writing vpm1675861-28 from 2515101 to 2515102 tid 3 2014-08-08T10:41:45.762 INFO:tasks.rados.rados.0.vpm167.stdout:3275: write oid 18 current snap is 432 2014-08-08T10:41:45.762 INFO:tasks.rados.rados.0.vpm167.stdout:3275: seq_num 1234 ranges {308506=1} 2014-08-08T10:41:45.765 INFO:tasks.rados.rados.0.vpm167.stdout:3275: writing vpm1675861-18 from 308506 to 308507 tid 1 2014-08-08T10:41:45.766 INFO:tasks.rados.rados.0.vpm167.stdout:3276: snap_remove snap 370 2014-08-08T10:41:46.005 INFO:tasks.rados.rados.0.vpm167.stdout:3274: finishing write tid 1 to vpm1675861-28 2014-08-08T10:41:46.010 INFO:tasks.rados.rados.0.vpm167.stdout:3274: finishing write tid 2 to vpm1675861-28 2014-08-08T10:41:46.013 INFO:tasks.rados.rados.0.vpm167.stdout:3274: finishing write tid 3 to vpm1675861-28 2014-08-08T10:41:46.013 INFO:tasks.rados.rados.0.vpm167.stdout:3274: finishing write tid 5 to vpm1675861-28 2014-08-08T10:41:46.026 INFO:tasks.rados.rados.0.vpm167.stdout:3275: finishing write tid 1 to vpm1675861-18 2014-08-08T10:41:46.026 INFO:tasks.rados.rados.0.vpm167.stdout:3274: finishing write tid 6 to vpm1675861-28 2014-08-08T10:41:46.026 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 761 (ObjNum 1233 snap 432 seq_num 1233) dirty exists 2014-08-08T10:41:46.086 INFO:tasks.rados.rados.0.vpm167.stdout:3275: finishing write tid 3 to vpm1675861-18 2014-08-08T10:41:46.299 INFO:tasks.rados.rados.0.vpm167.stdout:3275: finishing write tid 4 to vpm1675861-18 2014-08-08T10:41:46.300 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 18 v 2063 (ObjNum 1234 snap 432 seq_num 1234) dirty exists 2014-08-08T10:41:46.568 INFO:tasks.rados.rados.0.vpm167.stdout:3274: done (2 left) 2014-08-08T10:41:46.569 INFO:tasks.rados.rados.0.vpm167.stdout:3275: done (1 left) 2014-08-08T10:41:46.569 INFO:tasks.rados.rados.0.vpm167.stdout:3276: done (0 left) 2014-08-08T10:41:46.569 INFO:tasks.rados.rados.0.vpm167.stdout:3277: read oid 48 snap -1 2014-08-08T10:41:46.570 INFO:tasks.rados.rados.0.vpm167.stdout:3278: read oid 17 snap -1 2014-08-08T10:41:46.571 INFO:tasks.rados.rados.0.vpm167.stdout:3279: read oid 3 snap -1 2014-08-08T10:41:46.572 INFO:tasks.rados.rados.0.vpm167.stdout:3280: read oid 13 snap -1 2014-08-08T10:41:46.572 INFO:tasks.rados.rados.0.vpm167.stdout:3281: read oid 32 snap 415 2014-08-08T10:41:46.573 INFO:tasks.rados.rados.0.vpm167.stdout:3282: snap_remove snap 421 2014-08-08T10:41:46.700 INFO:tasks.rados.rados.0.vpm167.stdout:3279: expect (ObjNum 1231 snap 429 seq_num 1231) 2014-08-08T10:41:48.551 INFO:tasks.rados.rados.0.vpm167.stdout:3278: done (5 left) 2014-08-08T10:41:48.551 INFO:tasks.rados.rados.0.vpm167.stdout:3279: done (4 left) 2014-08-08T10:41:48.552 INFO:tasks.rados.rados.0.vpm167.stdout:3281: done (3 left) 2014-08-08T10:41:48.552 INFO:tasks.rados.rados.0.vpm167.stdout:3282: done (2 left) 2014-08-08T10:41:48.552 INFO:tasks.rados.rados.0.vpm167.stdout:3283: read oid 8 snap -1 2014-08-08T10:41:48.552 INFO:tasks.rados.rados.0.vpm167.stdout:3284: snap_remove snap 429 2014-08-08T10:41:48.553 INFO:tasks.rados.rados.0.vpm167.stdout:3277: expect (ObjNum 1210 snap 422 seq_num 1210) 2014-08-08T10:41:49.403 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:49.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:41:49.403 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.799514 2014-08-08T10:41:49.404 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 2 0.799513557429' 2014-08-08T10:41:50.112 INFO:tasks.rados.rados.0.vpm167.stdout:3280: expect (ObjNum 1216 snap 425 seq_num 1216) 2014-08-08T10:41:50.886 INFO:tasks.rados.rados.0.vpm167.stdout:3277: done (3 left) 2014-08-08T10:41:50.886 INFO:tasks.rados.rados.0.vpm167.stdout:3280: done (2 left) 2014-08-08T10:41:50.887 INFO:tasks.rados.rados.0.vpm167.stdout:3284: done (1 left) 2014-08-08T10:41:50.888 INFO:tasks.rados.rados.0.vpm167.stdout:3285: rollback oid 22 current snap is 432 2014-08-08T10:41:50.888 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 22 to 412 2014-08-08T10:41:50.888 INFO:tasks.rados.rados.0.vpm167.stdout:3286: write oid 23 current snap is 432 2014-08-08T10:41:50.889 INFO:tasks.rados.rados.0.vpm167.stdout:3286: seq_num 1235 ranges {3736=1} 2014-08-08T10:41:50.889 INFO:tasks.rados.rados.0.vpm167.stdout:3286: writing vpm1675861-23 from 3736 to 3737 tid 1 2014-08-08T10:41:50.890 INFO:tasks.rados.rados.0.vpm167.stdout:3287: read oid 49 snap -1 2014-08-08T10:41:50.890 INFO:tasks.rados.rados.0.vpm167.stdout:3288: write oid 29 current snap is 432 2014-08-08T10:41:50.891 INFO:tasks.rados.rados.0.vpm167.stdout:3288: seq_num 1236 ranges {216342=1} 2014-08-08T10:41:50.893 INFO:tasks.rados.rados.0.vpm167.stdout:3288: writing vpm1675861-29 from 216342 to 216343 tid 1 2014-08-08T10:41:50.894 INFO:tasks.rados.rados.0.vpm167.stdout:3283: expect (ObjNum 1063 snap 368 seq_num 1063) 2014-08-08T10:41:51.306 INFO:teuthology.orchestra.run.vpm166.stderr:set osd.2 primary-affinity to 0.799514 (8523962) 2014-08-08T10:41:52.292 INFO:tasks.rados.rados.0.vpm167.stdout:3283: done (4 left) 2014-08-08T10:41:52.293 INFO:tasks.rados.rados.0.vpm167.stdout:3289: rollback oid 43 current snap is 432 2014-08-08T10:41:52.293 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 43 to 423 2014-08-08T10:41:52.293 INFO:tasks.rados.rados.0.vpm167.stdout:3290: read oid 9 snap -1 2014-08-08T10:41:52.294 INFO:tasks.rados.rados.0.vpm167.stdout:3291: read oid 50 snap -1 2014-08-08T10:41:52.295 INFO:tasks.rados.rados.0.vpm167.stdout:3292: delete oid 28 current snap is 432 2014-08-08T10:41:52.295 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 22 v 1300 (ObjNum 46853200 snap 0 seq_num 47653456) dirty dne 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:3286: finishing write tid 1 to vpm1675861-23 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:3288: finishing write tid 1 to vpm1675861-29 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:3288: finishing write tid 3 to vpm1675861-29 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:3286: finishing write tid 3 to vpm1675861-23 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:3286: finishing write tid 4 to vpm1675861-23 2014-08-08T10:41:52.296 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 23 v 905 (ObjNum 1235 snap 432 seq_num 1235) dirty exists 2014-08-08T10:41:52.297 INFO:tasks.rados.rados.0.vpm167.stdout:3288: finishing write tid 4 to vpm1675861-29 2014-08-08T10:41:52.297 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 1113 (ObjNum 1236 snap 432 seq_num 1236) dirty exists 2014-08-08T10:41:52.297 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 0 (ObjNum 48210192 snap 0 seq_num 32) dirty dne 2014-08-08T10:41:52.316 INFO:tasks.rados.rados.0.vpm167.stdout:3291: expect (ObjNum 871 snap 293 seq_num 871) 2014-08-08T10:41:53.911 INFO:tasks.rados.rados.0.vpm167.stdout:3285: done (7 left) 2014-08-08T10:41:53.911 INFO:tasks.rados.rados.0.vpm167.stdout:3286: done (6 left) 2014-08-08T10:41:53.911 INFO:tasks.rados.rados.0.vpm167.stdout:3287: done (5 left) 2014-08-08T10:41:53.912 INFO:tasks.rados.rados.0.vpm167.stdout:3288: done (4 left) 2014-08-08T10:41:53.912 INFO:tasks.rados.rados.0.vpm167.stdout:3289: done (3 left) 2014-08-08T10:41:53.912 INFO:tasks.rados.rados.0.vpm167.stdout:3290: done (2 left) 2014-08-08T10:41:53.912 INFO:tasks.rados.rados.0.vpm167.stdout:3291: done (1 left) 2014-08-08T10:41:53.912 INFO:tasks.rados.rados.0.vpm167.stdout:3292: done (0 left) 2014-08-08T10:41:53.913 INFO:tasks.rados.rados.0.vpm167.stdout:3293: write oid 31 current snap is 432 2014-08-08T10:41:53.913 INFO:tasks.rados.rados.0.vpm167.stdout:3293: seq_num 1238 ranges {504978=438320,1433953=271025} 2014-08-08T10:41:53.930 INFO:tasks.rados.rados.0.vpm167.stdout:3293: writing vpm1675861-31 from 504978 to 943298 tid 1 2014-08-08T10:41:53.949 INFO:tasks.rados.rados.0.vpm167.stdout:3293: writing vpm1675861-31 from 1433953 to 1704978 tid 2 2014-08-08T10:41:53.951 INFO:tasks.rados.rados.0.vpm167.stdout:3294: snap_create 2014-08-08T10:41:53.952 INFO:tasks.rados.rados.0.vpm167.stdout:3293: finishing write tid 1 to vpm1675861-31 2014-08-08T10:41:54.846 INFO:tasks.rados.rados.0.vpm167.stdout:3293: finishing write tid 2 to vpm1675861-31 2014-08-08T10:41:54.846 INFO:tasks.rados.rados.0.vpm167.stdout:3293: finishing write tid 4 to vpm1675861-31 2014-08-08T10:41:54.850 INFO:tasks.rados.rados.0.vpm167.stdout:3293: finishing write tid 5 to vpm1675861-31 2014-08-08T10:41:54.850 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1304 (ObjNum 1238 snap 432 seq_num 1238) dirty exists 2014-08-08T10:41:54.951 INFO:tasks.rados.rados.0.vpm167.stdout:3293: done (1 left) 2014-08-08T10:41:54.951 INFO:tasks.rados.rados.0.vpm167.stdout:3294: done (0 left) 2014-08-08T10:41:54.952 INFO:tasks.rados.rados.0.vpm167.stdout:3295: read oid 16 snap 397 2014-08-08T10:41:54.952 INFO:tasks.rados.rados.0.vpm167.stdout:3296: snap_remove snap 424 2014-08-08T10:41:54.999 INFO:tasks.rados.rados.0.vpm167.stdout:3295: expect (ObjNum 1106 snap 391 seq_num 1106) 2014-08-08T10:41:56.006 INFO:tasks.rados.rados.0.vpm167.stdout:3295: done (1 left) 2014-08-08T10:41:56.006 INFO:tasks.rados.rados.0.vpm167.stdout:3296: done (0 left) 2014-08-08T10:41:56.007 INFO:tasks.rados.rados.0.vpm167.stdout:3297: snap_remove snap 399 2014-08-08T10:41:56.335 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:41:56.335 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T10:41:57.034 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T10:41:57.039 INFO:tasks.rados.rados.0.vpm167.stdout:3297: done (0 left) 2014-08-08T10:41:57.039 INFO:tasks.rados.rados.0.vpm167.stdout:3298: write oid 33 current snap is 433 2014-08-08T10:41:57.039 INFO:tasks.rados.rados.0.vpm167.stdout:3298: seq_num 1239 ranges {601232=694796,1401231=1} 2014-08-08T10:41:57.082 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-08T10:41:57.083 INFO:tasks.rados.rados.0.vpm167.stdout:3298: writing vpm1675861-33 from 601232 to 1296028 tid 1 2014-08-08T10:41:57.089 INFO:tasks.rados.rados.0.vpm167.stdout:3298: writing vpm1675861-33 from 1401231 to 1401232 tid 2 2014-08-08T10:41:57.089 INFO:tasks.rados.rados.0.vpm167.stdout:3299: rollback oid 27 current snap is 433 2014-08-08T10:41:57.090 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 27 to 432 2014-08-08T10:41:57.090 INFO:tasks.rados.rados.0.vpm167.stdout:3300: rollback oid 11 current snap is 433 2014-08-08T10:41:57.091 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 11 to 410 2014-08-08T10:41:57.091 INFO:tasks.rados.rados.0.vpm167.stdout:3301: write oid 34 current snap is 433 2014-08-08T10:41:57.091 INFO:tasks.rados.rados.0.vpm167.stdout:3301: seq_num 1240 ranges {509966=418025,1724298=728478,3206654=503312} 2014-08-08T10:41:57.119 INFO:tasks.rados.rados.0.vpm167.stdout:3301: writing vpm1675861-34 from 509966 to 927991 tid 1 2014-08-08T10:41:57.171 INFO:tasks.rados.rados.0.vpm167.stdout:3301: writing vpm1675861-34 from 1724298 to 2452776 tid 2 2014-08-08T10:41:57.208 INFO:tasks.rados.rados.0.vpm167.stdout:3301: writing vpm1675861-34 from 3206654 to 3709966 tid 3 2014-08-08T10:41:57.210 INFO:tasks.rados.rados.0.vpm167.stdout:3302: snap_create 2014-08-08T10:41:57.293 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 27 v 2179 (ObjNum 1114 snap 395 seq_num 1114) dirty exists 2014-08-08T10:41:57.372 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 1119 (ObjNum 1141 snap 404 seq_num 1141) dirty exists 2014-08-08T10:41:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:3298: finishing write tid 1 to vpm1675861-33 2014-08-08T10:41:57.813 INFO:tasks.rados.rados.0.vpm167.stdout:3298: finishing write tid 2 to vpm1675861-33 2014-08-08T10:41:57.839 INFO:tasks.rados.rados.0.vpm167.stdout:3301: finishing write tid 1 to vpm1675861-34 2014-08-08T10:41:57.839 INFO:tasks.rados.rados.0.vpm167.stdout:3298: finishing write tid 4 to vpm1675861-33 2014-08-08T10:41:57.841 INFO:tasks.rados.rados.0.vpm167.stdout:3301: finishing write tid 2 to vpm1675861-34 2014-08-08T10:41:57.846 INFO:tasks.rados.rados.0.vpm167.stdout:3301: finishing write tid 3 to vpm1675861-34 2014-08-08T10:41:57.873 INFO:tasks.rados.rados.0.vpm167.stdout:3301: finishing write tid 5 to vpm1675861-34 2014-08-08T10:41:57.931 INFO:tasks.rados.rados.0.vpm167.stdout:3298: finishing write tid 5 to vpm1675861-33 2014-08-08T10:41:57.931 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1761 (ObjNum 1239 snap 433 seq_num 1239) dirty exists 2014-08-08T10:41:57.931 INFO:tasks.rados.rados.0.vpm167.stdout:3301: finishing write tid 6 to vpm1675861-34 2014-08-08T10:41:57.932 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1769 (ObjNum 1240 snap 433 seq_num 1240) dirty exists 2014-08-08T10:41:58.870 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T10:41:58.895 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T10:41:59.062 INFO:tasks.rados.rados.0.vpm167.stdout:3298: done (4 left) 2014-08-08T10:41:59.063 INFO:tasks.rados.rados.0.vpm167.stdout:3299: done (3 left) 2014-08-08T10:41:59.064 INFO:tasks.rados.rados.0.vpm167.stdout:3300: done (2 left) 2014-08-08T10:41:59.064 INFO:tasks.rados.rados.0.vpm167.stdout:3301: done (1 left) 2014-08-08T10:41:59.064 INFO:tasks.rados.rados.0.vpm167.stdout:3302: done (0 left) 2014-08-08T10:41:59.064 INFO:tasks.rados.rados.0.vpm167.stdout:3303: delete oid 24 current snap is 434 2014-08-08T10:41:59.156 INFO:tasks.rados.rados.0.vpm167.stdout:3303: done (0 left) 2014-08-08T10:41:59.157 INFO:tasks.rados.rados.0.vpm167.stdout:3304: read oid 41 snap -1 2014-08-08T10:41:59.157 INFO:tasks.rados.rados.0.vpm167.stdout:3305: snap_create 2014-08-08T10:41:59.284 INFO:tasks.rados.rados.0.vpm167.stdout:3304: expect (ObjNum 1211 snap 423 seq_num 1211) 2014-08-08T10:42:00.214 INFO:tasks.rados.rados.0.vpm167.stdout:3304: done (1 left) 2014-08-08T10:42:00.215 INFO:tasks.rados.rados.0.vpm167.stdout:3305: done (0 left) 2014-08-08T10:42:00.215 INFO:tasks.rados.rados.0.vpm167.stdout:3306: snap_remove snap 386 2014-08-08T10:42:00.238 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T10:42:00.255 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T10:42:01.218 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T10:42:01.247 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T10:42:01.306 INFO:tasks.rados.rados.0.vpm167.stdout:3306: done (0 left) 2014-08-08T10:42:01.306 INFO:tasks.rados.rados.0.vpm167.stdout:3307: write oid 44 current snap is 435 2014-08-08T10:42:01.306 INFO:tasks.rados.rados.0.vpm167.stdout:3307: seq_num 1242 ranges {701449=514081,1952561=468868,3093775=7674} 2014-08-08T10:42:01.328 INFO:tasks.rados.rados.0.vpm167.stdout:3307: writing vpm1675861-44 from 701449 to 1215530 tid 1 2014-08-08T10:42:01.360 INFO:tasks.rados.rados.0.vpm167.stdout:3307: writing vpm1675861-44 from 1952561 to 2421429 tid 2 2014-08-08T10:42:01.368 INFO:tasks.rados.rados.0.vpm167.stdout:3307: writing vpm1675861-44 from 3093775 to 3101449 tid 3 2014-08-08T10:42:01.369 INFO:tasks.rados.rados.0.vpm167.stdout:3308: write oid 14 current snap is 435 2014-08-08T10:42:01.370 INFO:tasks.rados.rados.0.vpm167.stdout:3308: seq_num 1243 ranges {645255=769534,1914246=574135,3178649=666606} 2014-08-08T10:42:01.400 INFO:tasks.rados.rados.0.vpm167.stdout:3308: writing vpm1675861-14 from 645255 to 1414789 tid 1 2014-08-08T10:42:01.424 INFO:tasks.rados.rados.0.vpm167.stdout:3308: writing vpm1675861-14 from 1914246 to 2488381 tid 2 2014-08-08T10:42:01.452 INFO:tasks.rados.rados.0.vpm167.stdout:3308: writing vpm1675861-14 from 3178649 to 3845255 tid 3 2014-08-08T10:42:01.454 INFO:tasks.rados.rados.0.vpm167.stdout:3309: read oid 10 snap 432 2014-08-08T10:42:01.454 INFO:tasks.rados.rados.0.vpm167.stdout:3310: delete oid 43 current snap is 435 2014-08-08T10:42:01.465 INFO:tasks.rados.rados.0.vpm167.stdout:3309: expect (ObjNum 1227 snap 428 seq_num 1227) 2014-08-08T10:42:01.621 INFO:tasks.rados.rados.0.vpm167.stdout:3309: done (3 left) 2014-08-08T10:42:01.621 INFO:tasks.rados.rados.0.vpm167.stdout:3310: done (2 left) 2014-08-08T10:42:01.621 INFO:tasks.rados.rados.0.vpm167.stdout:3311: write oid 17 current snap is 435 2014-08-08T10:42:01.622 INFO:tasks.rados.rados.0.vpm167.stdout:3311: seq_num 1245 ranges {402690=400000} 2014-08-08T10:42:01.644 INFO:tasks.rados.rados.0.vpm167.stdout:3311: writing vpm1675861-17 from 402690 to 802690 tid 1 2014-08-08T10:42:01.650 INFO:tasks.rados.rados.0.vpm167.stdout:3312: read oid 48 snap -1 2014-08-08T10:42:01.650 INFO:tasks.rados.rados.0.vpm167.stdout:3313: rollback oid 20 current snap is 435 2014-08-08T10:42:01.650 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 20 to 415 2014-08-08T10:42:01.650 INFO:tasks.rados.rados.0.vpm167.stdout:3314: read oid 34 snap 396 2014-08-08T10:42:01.650 INFO:tasks.rados.rados.0.vpm167.stdout:3315: snap_create 2014-08-08T10:42:01.875 INFO:tasks.rados.rados.0.vpm167.stdout:3311: finishing write tid 1 to vpm1675861-17 2014-08-08T10:42:01.950 INFO:tasks.rados.rados.0.vpm167.stdout:3308: finishing write tid 1 to vpm1675861-14 2014-08-08T10:42:02.066 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 20 v 945 (ObjNum 1041 snap 358 seq_num 1041) dirty exists 2014-08-08T10:42:02.093 INFO:tasks.rados.rados.0.vpm167.stdout:3307: finishing write tid 1 to vpm1675861-44 2014-08-08T10:42:02.137 INFO:tasks.rados.rados.0.vpm167.stdout:3311: finishing write tid 3 to vpm1675861-17 2014-08-08T10:42:02.141 INFO:tasks.rados.rados.0.vpm167.stdout:3307: finishing write tid 2 to vpm1675861-44 2014-08-08T10:42:02.141 INFO:tasks.rados.rados.0.vpm167.stdout:3307: finishing write tid 3 to vpm1675861-44 2014-08-08T10:42:02.329 INFO:tasks.rados.rados.0.vpm167.stdout:3307: finishing write tid 5 to vpm1675861-44 2014-08-08T10:42:02.502 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T10:42:02.527 INFO:tasks.rados.rados.0.vpm167.stdout:3308: finishing write tid 2 to vpm1675861-14 2014-08-08T10:42:02.528 INFO:tasks.rados.rados.0.vpm167.stdout:3308: finishing write tid 3 to vpm1675861-14 2014-08-08T10:42:02.529 INFO:tasks.rados.rados.0.vpm167.stdout:3308: finishing write tid 5 to vpm1675861-14 2014-08-08T10:42:02.551 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T10:42:02.551 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:42:02.579 INFO:tasks.rados.rados.0.vpm167.stdout:3308: finishing write tid 6 to vpm1675861-14 2014-08-08T10:42:02.580 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2090 (ObjNum 1243 snap 435 seq_num 1243) dirty exists 2014-08-08T10:42:02.673 INFO:tasks.rados.rados.0.vpm167.stdout:3308: done (6 left) 2014-08-08T10:42:02.674 INFO:tasks.rados.rados.0.vpm167.stdout:3313: done (5 left) 2014-08-08T10:42:02.674 INFO:tasks.rados.rados.0.vpm167.stdout:3314: done (4 left) 2014-08-08T10:42:02.674 INFO:tasks.rados.rados.0.vpm167.stdout:3315: done (3 left) 2014-08-08T10:42:02.675 INFO:tasks.rados.rados.0.vpm167.stdout:3316: snap_create 2014-08-08T10:42:02.675 INFO:tasks.rados.rados.0.vpm167.stdout:3311: finishing write tid 4 to vpm1675861-17 2014-08-08T10:42:02.676 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2190 (ObjNum 1245 snap 435 seq_num 1245) dirty exists 2014-08-08T10:42:02.731 INFO:tasks.rados.rados.0.vpm167.stdout:3307: finishing write tid 6 to vpm1675861-44 2014-08-08T10:42:02.731 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1451 (ObjNum 1242 snap 435 seq_num 1242) dirty exists 2014-08-08T10:42:03.220 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:42:03.280 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:42:03.525 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:42:03.605 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:42:03.924 INFO:tasks.rados.rados.0.vpm167.stdout:3307: done (3 left) 2014-08-08T10:42:03.924 INFO:tasks.rados.rados.0.vpm167.stdout:3311: done (2 left) 2014-08-08T10:42:03.925 INFO:tasks.rados.rados.0.vpm167.stdout:3316: done (1 left) 2014-08-08T10:42:03.925 INFO:tasks.rados.rados.0.vpm167.stdout:3317: read oid 24 snap -1 2014-08-08T10:42:03.926 INFO:tasks.rados.rados.0.vpm167.stdout:3318: write oid 45 current snap is 437 2014-08-08T10:42:03.926 INFO:tasks.rados.rados.0.vpm167.stdout:3318: seq_num 1246 ranges {746496=588819,1546495=1} 2014-08-08T10:42:03.960 INFO:tasks.rados.rados.0.vpm167.stdout:3318: writing vpm1675861-45 from 746496 to 1335315 tid 1 2014-08-08T10:42:03.973 INFO:tasks.rados.rados.0.vpm167.stdout:3318: writing vpm1675861-45 from 1546495 to 1546496 tid 2 2014-08-08T10:42:03.975 INFO:tasks.rados.rados.0.vpm167.stdout:3319: write oid 33 current snap is 437 2014-08-08T10:42:03.975 INFO:tasks.rados.rados.0.vpm167.stdout:3319: seq_num 1247 ranges {691326=560623,1899606=391720} 2014-08-08T10:42:04.011 INFO:tasks.rados.rados.0.vpm167.stdout:3319: writing vpm1675861-33 from 691326 to 1251949 tid 1 2014-08-08T10:42:04.042 INFO:tasks.rados.rados.0.vpm167.stdout:3319: writing vpm1675861-33 from 1899606 to 2291326 tid 2 2014-08-08T10:42:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3317: done (3 left) 2014-08-08T10:42:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3320: snap_create 2014-08-08T10:42:04.074 INFO:tasks.rados.rados.0.vpm167.stdout:3318: finishing write tid 1 to vpm1675861-45 2014-08-08T10:42:04.106 INFO:tasks.rados.rados.0.vpm167.stdout:3318: finishing write tid 2 to vpm1675861-45 2014-08-08T10:42:04.106 INFO:tasks.rados.rados.0.vpm167.stdout:3318: finishing write tid 4 to vpm1675861-45 2014-08-08T10:42:04.107 INFO:tasks.rados.rados.0.vpm167.stdout:3318: finishing write tid 5 to vpm1675861-45 2014-08-08T10:42:04.107 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 1461 (ObjNum 1246 snap 437 seq_num 1246) dirty exists 2014-08-08T10:42:04.357 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:41:19.622013', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88518744, u'health': u'HEALTH_OK', u'kb_used': 10434824, u'store_stats': {u'bytes_total': 27826286, u'bytes_log': 972874, u'last_updated': u'0.000000', u'bytes_misc': 26853412, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:41:26.392994', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87432196, u'health': u'HEALTH_OK', u'kb_used': 11521372, u'store_stats': {u'bytes_total': 28898660, u'bytes_log': 2059276, u'last_updated': u'0.000000', u'bytes_misc': 26839384, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:41:23.351859', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88513976, u'health': u'HEALTH_OK', u'kb_used': 10439592, u'store_stats': {u'bytes_total': 28588291, u'bytes_log': 1728789, u'last_updated': u'0.000000', u'bytes_misc': 26859502, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'num_pgs': 338, u'data_bytes': 274521500, u'bytes_used': 1570336768, u'version': 2373, u'pgs_by_state': [{u'count': 338, u'state_name': u'active+clean'}], u'bytes_avail': 1071647199232}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1277, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:42:04.358 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T10:42:04.358 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:04.359 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.794506526338 2014-08-08T10:42:04.359 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 0.794506526338' 2014-08-08T10:42:05.862 INFO:tasks.rados.rados.0.vpm167.stdout:3318: done (3 left) 2014-08-08T10:42:05.863 INFO:tasks.rados.rados.0.vpm167.stdout:3320: done (2 left) 2014-08-08T10:42:05.863 INFO:tasks.rados.rados.0.vpm167.stdout:3321: write oid 42 current snap is 438 2014-08-08T10:42:05.863 INFO:tasks.rados.rados.0.vpm167.stdout:3321: seq_num 1248 ranges {786556=416075,1861291=125265} 2014-08-08T10:42:05.892 INFO:tasks.rados.rados.0.vpm167.stdout:3321: writing vpm1675861-42 from 786556 to 1202631 tid 1 2014-08-08T10:42:05.909 INFO:tasks.rados.rados.0.vpm167.stdout:3321: writing vpm1675861-42 from 1861291 to 1986556 tid 2 2014-08-08T10:42:05.911 INFO:tasks.rados.rados.0.vpm167.stdout:3322: delete oid 36 current snap is 438 2014-08-08T10:42:06.043 INFO:tasks.rados.rados.0.vpm167.stdout:3322: done (3 left) 2014-08-08T10:42:06.043 INFO:tasks.rados.rados.0.vpm167.stdout:3323: read oid 2 snap -1 2014-08-08T10:42:06.044 INFO:tasks.rados.rados.0.vpm167.stdout:3324: write oid 13 current snap is 438 2014-08-08T10:42:06.044 INFO:tasks.rados.rados.0.vpm167.stdout:3324: seq_num 1250 ranges {791543=512132,1770835=568350,2897883=761046,3991542=1} 2014-08-08T10:42:06.080 INFO:tasks.rados.rados.0.vpm167.stdout:3324: writing vpm1675861-13 from 791543 to 1303675 tid 1 2014-08-08T10:42:06.105 INFO:tasks.rados.rados.0.vpm167.stdout:3324: writing vpm1675861-13 from 1770835 to 2339185 tid 2 2014-08-08T10:42:06.139 INFO:tasks.rados.rados.0.vpm167.stdout:3324: writing vpm1675861-13 from 2897883 to 3658929 tid 3 2014-08-08T10:42:06.144 INFO:tasks.rados.rados.0.vpm167.stdout:3324: writing vpm1675861-13 from 3991542 to 3991543 tid 4 2014-08-08T10:42:06.145 INFO:tasks.rados.rados.0.vpm167.stdout:3323: expect (ObjNum 1101 snap 385 seq_num 1101) 2014-08-08T10:42:07.331 INFO:tasks.rados.rados.0.vpm167.stdout:3323: done (4 left) 2014-08-08T10:42:07.331 INFO:tasks.rados.rados.0.vpm167.stdout:3325: read oid 14 snap -1 2014-08-08T10:42:07.331 INFO:tasks.rados.rados.0.vpm167.stdout:3326: rollback oid 27 current snap is 438 2014-08-08T10:42:07.332 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 27 to 397 2014-08-08T10:42:07.332 INFO:tasks.rados.rados.0.vpm167.stdout:3327: delete oid 40 current snap is 438 2014-08-08T10:42:07.395 INFO:tasks.rados.rados.0.vpm167.stdout:3325: expect (ObjNum 1243 snap 435 seq_num 1243) 2014-08-08T10:42:08.077 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.2 to 0.794507 (8520682) 2014-08-08T10:42:09.545 INFO:tasks.rados.rados.0.vpm167.stdout:3325: done (6 left) 2014-08-08T10:42:09.545 INFO:tasks.rados.rados.0.vpm167.stdout:3327: done (5 left) 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:3328: snap_remove snap 412 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:3321: finishing write tid 1 to vpm1675861-42 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:3321: finishing write tid 2 to vpm1675861-42 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:3321: finishing write tid 4 to vpm1675861-42 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:3321: finishing write tid 5 to vpm1675861-42 2014-08-08T10:42:09.546 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 2098 (ObjNum 1248 snap 438 seq_num 1248) dirty exists 2014-08-08T10:42:09.685 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 27 v 2192 (ObjNum 1114 snap 395 seq_num 1114) dirty exists 2014-08-08T10:42:09.825 INFO:tasks.rados.rados.0.vpm167.stdout:3319: finishing write tid 1 to vpm1675861-33 2014-08-08T10:42:09.825 INFO:tasks.rados.rados.0.vpm167.stdout:3319: finishing write tid 2 to vpm1675861-33 2014-08-08T10:42:09.825 INFO:tasks.rados.rados.0.vpm167.stdout:3319: finishing write tid 4 to vpm1675861-33 2014-08-08T10:42:10.128 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 1 to vpm1675861-13 2014-08-08T10:42:10.148 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 2 to vpm1675861-13 2014-08-08T10:42:10.148 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 3 to vpm1675861-13 2014-08-08T10:42:10.148 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 4 to vpm1675861-13 2014-08-08T10:42:10.149 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 6 to vpm1675861-13 2014-08-08T10:42:10.275 INFO:tasks.rados.rados.0.vpm167.stdout:3321: done (5 left) 2014-08-08T10:42:10.276 INFO:tasks.rados.rados.0.vpm167.stdout:3326: done (4 left) 2014-08-08T10:42:10.276 INFO:tasks.rados.rados.0.vpm167.stdout:3328: done (3 left) 2014-08-08T10:42:10.276 INFO:tasks.rados.rados.0.vpm167.stdout:3329: snap_remove snap 431 2014-08-08T10:42:10.471 INFO:tasks.rados.rados.0.vpm167.stdout:3319: finishing write tid 5 to vpm1675861-33 2014-08-08T10:42:10.472 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1773 (ObjNum 1247 snap 437 seq_num 1247) dirty exists 2014-08-08T10:42:11.310 INFO:tasks.rados.rados.0.vpm167.stdout:3319: done (3 left) 2014-08-08T10:42:11.310 INFO:tasks.rados.rados.0.vpm167.stdout:3329: done (2 left) 2014-08-08T10:42:11.311 INFO:tasks.rados.rados.0.vpm167.stdout:3330: read oid 8 snap -1 2014-08-08T10:42:11.312 INFO:tasks.rados.rados.0.vpm167.stdout:3331: rollback oid 21 current snap is 438 2014-08-08T10:42:11.312 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 21 to 410 2014-08-08T10:42:11.313 INFO:tasks.rados.rados.0.vpm167.stdout:3332: write oid 28 current snap is 438 2014-08-08T10:42:11.313 INFO:tasks.rados.rados.0.vpm167.stdout:3332: seq_num 1252 ranges {431602=1} 2014-08-08T10:42:11.315 INFO:tasks.rados.rados.0.vpm167.stdout:3332: writing vpm1675861-28 from 431602 to 431603 tid 1 2014-08-08T10:42:11.317 INFO:tasks.rados.rados.0.vpm167.stdout:3333: write oid 14 current snap is 438 2014-08-08T10:42:11.317 INFO:tasks.rados.rados.0.vpm167.stdout:3333: seq_num 1253 ranges {775408=400000} 2014-08-08T10:42:11.339 INFO:tasks.rados.rados.0.vpm167.stdout:3333: writing vpm1675861-14 from 775408 to 1175408 tid 1 2014-08-08T10:42:11.343 INFO:tasks.rados.rados.0.vpm167.stdout:3334: read oid 2 snap -1 2014-08-08T10:42:11.344 INFO:tasks.rados.rados.0.vpm167.stdout:3335: write oid 7 current snap is 438 2014-08-08T10:42:11.344 INFO:tasks.rados.rados.0.vpm167.stdout:3335: seq_num 1254 ranges {588014=587894,1388013=1} 2014-08-08T10:42:11.368 INFO:tasks.rados.rados.0.vpm167.stdout:3335: writing vpm1675861-7 from 588014 to 1175908 tid 1 2014-08-08T10:42:11.371 INFO:tasks.rados.rados.0.vpm167.stdout:3335: writing vpm1675861-7 from 1388013 to 1388014 tid 2 2014-08-08T10:42:11.371 INFO:tasks.rados.rados.0.vpm167.stdout:3336: write oid 47 current snap is 438 2014-08-08T10:42:11.372 INFO:tasks.rados.rados.0.vpm167.stdout:3336: seq_num 1255 ranges {532844=559698,1718904=413940} 2014-08-08T10:42:11.393 INFO:tasks.rados.rados.0.vpm167.stdout:3336: writing vpm1675861-47 from 532844 to 1092542 tid 1 2014-08-08T10:42:11.414 INFO:tasks.rados.rados.0.vpm167.stdout:3336: writing vpm1675861-47 from 1718904 to 2132844 tid 2 2014-08-08T10:42:11.416 INFO:tasks.rados.rados.0.vpm167.stdout:3337: rollback oid 34 current snap is 438 2014-08-08T10:42:11.416 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 34 to 415 2014-08-08T10:42:11.416 INFO:tasks.rados.rados.0.vpm167.stdout:3338: read oid 15 snap -1 2014-08-08T10:42:11.416 INFO:tasks.rados.rados.0.vpm167.stdout:3339: read oid 4 snap -1 2014-08-08T10:42:11.417 INFO:tasks.rados.rados.0.vpm167.stdout:3340: read oid 19 snap 419 2014-08-08T10:42:11.417 INFO:tasks.rados.rados.0.vpm167.stdout:3341: read oid 17 snap 416 2014-08-08T10:42:11.417 INFO:tasks.rados.rados.0.vpm167.stdout:3342: read oid 22 snap 436 2014-08-08T10:42:11.417 INFO:tasks.rados.rados.0.vpm167.stdout:3343: delete oid 43 current snap is 438 2014-08-08T10:42:11.473 INFO:tasks.rados.rados.0.vpm167.stdout:3334: expect (ObjNum 1101 snap 385 seq_num 1101) 2014-08-08T10:42:13.011 INFO:tasks.rados.rados.0.vpm167.stdout:3334: done (15 left) 2014-08-08T10:42:13.011 INFO:tasks.rados.rados.0.vpm167.stdout:3342: done (14 left) 2014-08-08T10:42:13.011 INFO:tasks.rados.rados.0.vpm167.stdout:3343: done (13 left) 2014-08-08T10:42:13.011 INFO:tasks.rados.rados.0.vpm167.stdout:3344: read oid 24 snap -1 2014-08-08T10:42:13.012 INFO:tasks.rados.rados.0.vpm167.stdout:3345: delete oid 5 current snap is 438 2014-08-08T10:42:13.013 INFO:tasks.rados.rados.0.vpm167.stdout:3339: expect (ObjNum 1221 snap 427 seq_num 1221) 2014-08-08T10:42:13.131 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:13.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:13.132 INFO:tasks.thrashosds.thrasher:Growing pool data 2014-08-08T10:42:13.132 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:42:13.622 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:42:13.681 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T10:42:13.682 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 54' 2014-08-08T10:42:13.782 INFO:tasks.rados.rados.0.vpm167.stdout:3339: done (14 left) 2014-08-08T10:42:13.782 INFO:tasks.rados.rados.0.vpm167.stdout:3345: done (13 left) 2014-08-08T10:42:13.783 INFO:tasks.rados.rados.0.vpm167.stdout:3346: snap_remove snap 420 2014-08-08T10:42:13.783 INFO:tasks.rados.rados.0.vpm167.stdout:3340: expect (ObjNum 1197 snap 419 seq_num 1197) 2014-08-08T10:42:14.286 INFO:tasks.rados.rados.0.vpm167.stdout:3324: finishing write tid 7 to vpm1675861-13 2014-08-08T10:42:14.286 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1779 (ObjNum 1250 snap 438 seq_num 1250) dirty exists 2014-08-08T10:42:14.286 INFO:tasks.rados.rados.0.vpm167.stdout:3338: expect (ObjNum 1213 snap 423 seq_num 1213) 2014-08-08T10:42:14.457 INFO:tasks.rados.rados.0.vpm167.stdout:3341: expect (ObjNum 1180 snap 415 seq_num 1180) 2014-08-08T10:42:14.521 INFO:tasks.rados.rados.0.vpm167.stdout:3333: finishing write tid 1 to vpm1675861-14 2014-08-08T10:42:14.522 INFO:tasks.rados.rados.0.vpm167.stdout:3333: finishing write tid 3 to vpm1675861-14 2014-08-08T10:42:14.522 INFO:tasks.rados.rados.0.vpm167.stdout:3333: finishing write tid 4 to vpm1675861-14 2014-08-08T10:42:14.522 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2105 (ObjNum 1253 snap 438 seq_num 1253) dirty exists 2014-08-08T10:42:14.522 INFO:tasks.rados.rados.0.vpm167.stdout:3335: finishing write tid 1 to vpm1675861-7 2014-08-08T10:42:14.523 INFO:tasks.rados.rados.0.vpm167.stdout:3335: finishing write tid 2 to vpm1675861-7 2014-08-08T10:42:14.523 INFO:tasks.rados.rados.0.vpm167.stdout:3335: finishing write tid 4 to vpm1675861-7 2014-08-08T10:42:14.523 INFO:tasks.rados.rados.0.vpm167.stdout:3335: finishing write tid 5 to vpm1675861-7 2014-08-08T10:42:14.523 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 826 (ObjNum 1254 snap 438 seq_num 1254) dirty exists 2014-08-08T10:42:14.524 INFO:tasks.rados.rados.0.vpm167.stdout:3332: finishing write tid 1 to vpm1675861-28 2014-08-08T10:42:14.524 INFO:tasks.rados.rados.0.vpm167.stdout:3332: finishing write tid 3 to vpm1675861-28 2014-08-08T10:42:14.524 INFO:tasks.rados.rados.0.vpm167.stdout:3312: expect (ObjNum 1210 snap 422 seq_num 1210) 2014-08-08T10:42:14.682 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 0 pg_num to 54 2014-08-08T10:42:16.529 INFO:tasks.rados.rados.0.vpm167.stdout:3312: done (13 left) 2014-08-08T10:42:16.529 INFO:tasks.rados.rados.0.vpm167.stdout:3324: done (12 left) 2014-08-08T10:42:16.530 INFO:tasks.rados.rados.0.vpm167.stdout:3333: done (11 left) 2014-08-08T10:42:16.530 INFO:tasks.rados.rados.0.vpm167.stdout:3335: done (10 left) 2014-08-08T10:42:16.530 INFO:tasks.rados.rados.0.vpm167.stdout:3338: done (9 left) 2014-08-08T10:42:16.530 INFO:tasks.rados.rados.0.vpm167.stdout:3340: done (8 left) 2014-08-08T10:42:16.531 INFO:tasks.rados.rados.0.vpm167.stdout:3341: done (7 left) 2014-08-08T10:42:16.531 INFO:tasks.rados.rados.0.vpm167.stdout:3344: done (6 left) 2014-08-08T10:42:16.531 INFO:tasks.rados.rados.0.vpm167.stdout:3346: done (5 left) 2014-08-08T10:42:16.532 INFO:tasks.rados.rados.0.vpm167.stdout:3347: read oid 10 snap -1 2014-08-08T10:42:16.542 INFO:tasks.rados.rados.0.vpm167.stdout:3348: read oid 16 snap -1 2014-08-08T10:42:16.542 INFO:tasks.rados.rados.0.vpm167.stdout:3349: read oid 18 snap -1 2014-08-08T10:42:16.542 INFO:tasks.rados.rados.0.vpm167.stdout:3350: delete oid 24 current snap is 438 2014-08-08T10:42:16.543 INFO:tasks.rados.rados.0.vpm167.stdout:3330: expect (ObjNum 1063 snap 368 seq_num 1063) 2014-08-08T10:42:18.337 INFO:tasks.rados.rados.0.vpm167.stdout:3330: done (8 left) 2014-08-08T10:42:18.338 INFO:tasks.rados.rados.0.vpm167.stdout:3350: done (7 left) 2014-08-08T10:42:18.339 INFO:tasks.rados.rados.0.vpm167.stdout:3351: snap_create 2014-08-08T10:42:18.339 INFO:tasks.rados.rados.0.vpm167.stdout:3332: finishing write tid 4 to vpm1675861-28 2014-08-08T10:42:18.340 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 780 (ObjNum 1252 snap 438 seq_num 1252) dirty exists 2014-08-08T10:42:18.340 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 21 v 422 (ObjNum 1010 snap 345 seq_num 1010) dirty exists 2014-08-08T10:42:18.341 INFO:tasks.rados.rados.0.vpm167.stdout:3336: finishing write tid 1 to vpm1675861-47 2014-08-08T10:42:18.341 INFO:tasks.rados.rados.0.vpm167.stdout:3336: finishing write tid 2 to vpm1675861-47 2014-08-08T10:42:18.341 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 34 v 1782 (ObjNum 31671728 snap 0 seq_num 0) dirty dne 2014-08-08T10:42:18.342 INFO:tasks.rados.rados.0.vpm167.stdout:3336: finishing write tid 4 to vpm1675861-47 2014-08-08T10:42:18.342 INFO:tasks.rados.rados.0.vpm167.stdout:3336: finishing write tid 5 to vpm1675861-47 2014-08-08T10:42:18.342 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 47 v 1372 (ObjNum 1255 snap 438 seq_num 1255) dirty exists 2014-08-08T10:42:18.342 INFO:tasks.rados.rados.0.vpm167.stdout:3347: expect (ObjNum 1227 snap 428 seq_num 1227) 2014-08-08T10:42:18.553 INFO:tasks.rados.rados.0.vpm167.stdout:3331: done (7 left) 2014-08-08T10:42:18.553 INFO:tasks.rados.rados.0.vpm167.stdout:3332: done (6 left) 2014-08-08T10:42:18.554 INFO:tasks.rados.rados.0.vpm167.stdout:3336: done (5 left) 2014-08-08T10:42:18.554 INFO:tasks.rados.rados.0.vpm167.stdout:3337: done (4 left) 2014-08-08T10:42:18.554 INFO:tasks.rados.rados.0.vpm167.stdout:3347: done (3 left) 2014-08-08T10:42:18.554 INFO:tasks.rados.rados.0.vpm167.stdout:3348: done (2 left) 2014-08-08T10:42:18.555 INFO:tasks.rados.rados.0.vpm167.stdout:3351: done (1 left) 2014-08-08T10:42:18.555 INFO:tasks.rados.rados.0.vpm167.stdout:3352: delete oid 43 current snap is 439 2014-08-08T10:42:18.555 INFO:tasks.rados.rados.0.vpm167.stdout:3349: expect (ObjNum 1234 snap 432 seq_num 1234) 2014-08-08T10:42:18.591 INFO:tasks.rados.rados.0.vpm167.stdout:3349: done (1 left) 2014-08-08T10:42:18.591 INFO:tasks.rados.rados.0.vpm167.stdout:3352: done (0 left) 2014-08-08T10:42:18.591 INFO:tasks.rados.rados.0.vpm167.stdout:3353: snap_create 2014-08-08T10:42:19.508 INFO:tasks.rados.rados.0.vpm167.stdout:3353: done (0 left) 2014-08-08T10:42:19.508 INFO:tasks.rados.rados.0.vpm167.stdout:3354: snap_remove snap 423 2014-08-08T10:42:19.713 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:19.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:19.714 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T10:42:19.714 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:42:19.714 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:42:19.715 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T10:42:20.771 INFO:tasks.rados.rados.0.vpm167.stdout:3354: done (0 left) 2014-08-08T10:42:20.771 INFO:tasks.rados.rados.0.vpm167.stdout:3355: snap_remove snap 430 2014-08-08T10:42:21.868 INFO:tasks.rados.rados.0.vpm167.stdout:3355: done (0 left) 2014-08-08T10:42:21.869 INFO:tasks.rados.rados.0.vpm167.stdout:3356: read oid 43 snap -1 2014-08-08T10:42:21.870 INFO:tasks.rados.rados.0.vpm167.stdout:3357: write oid 4 current snap is 440 2014-08-08T10:42:21.870 INFO:tasks.rados.rados.0.vpm167.stdout:3357: seq_num 1260 ranges {273119=1} 2014-08-08T10:42:21.871 INFO:tasks.rados.rados.0.vpm167.stdout:3357: writing vpm1675861-4 from 273119 to 273120 tid 1 2014-08-08T10:42:21.872 INFO:tasks.rados.rados.0.vpm167.stdout:3358: write oid 45 current snap is 440 2014-08-08T10:42:21.872 INFO:tasks.rados.rados.0.vpm167.stdout:3358: seq_num 1261 ranges {616926=400000} 2014-08-08T10:42:21.891 INFO:tasks.rados.rados.0.vpm167.stdout:3358: writing vpm1675861-45 from 616926 to 1016926 tid 1 2014-08-08T10:42:21.894 INFO:tasks.rados.rados.0.vpm167.stdout:3356: done (2 left) 2014-08-08T10:42:21.894 INFO:tasks.rados.rados.0.vpm167.stdout:3359: write oid 41 current snap is 440 2014-08-08T10:42:21.894 INFO:tasks.rados.rados.0.vpm167.stdout:3359: seq_num 1262 ranges {713179=1} 2014-08-08T10:42:21.901 INFO:tasks.rados.rados.0.vpm167.stdout:3359: writing vpm1675861-41 from 713179 to 713180 tid 1 2014-08-08T10:42:21.902 INFO:tasks.rados.rados.0.vpm167.stdout:3360: read oid 19 snap -1 2014-08-08T10:42:21.903 INFO:tasks.rados.rados.0.vpm167.stdout:3361: read oid 29 snap 419 2014-08-08T10:42:21.903 INFO:tasks.rados.rados.0.vpm167.stdout:3362: write oid 17 current snap is 440 2014-08-08T10:42:21.904 INFO:tasks.rados.rados.0.vpm167.stdout:3362: seq_num 1263 ranges {773338=557749,1971225=2113} 2014-08-08T10:42:21.927 INFO:tasks.rados.rados.0.vpm167.stdout:3362: writing vpm1675861-17 from 773338 to 1331087 tid 1 2014-08-08T10:42:21.940 INFO:tasks.rados.rados.0.vpm167.stdout:3362: writing vpm1675861-17 from 1971225 to 1973338 tid 2 2014-08-08T10:42:21.942 INFO:tasks.rados.rados.0.vpm167.stdout:3361: expect (ObjNum 1153 snap 407 seq_num 1153) 2014-08-08T10:42:21.950 INFO:tasks.rados.rados.0.vpm167.stdout:3361: done (5 left) 2014-08-08T10:42:21.951 INFO:tasks.rados.rados.0.vpm167.stdout:3363: write oid 43 current snap is 440 2014-08-08T10:42:21.951 INFO:tasks.rados.rados.0.vpm167.stdout:3363: seq_num 1264 ranges {718167=565649,1818604=473450,2718166=1} 2014-08-08T10:42:21.977 INFO:tasks.rados.rados.0.vpm167.stdout:3363: writing vpm1675861-43 from 718167 to 1283816 tid 1 2014-08-08T10:42:22.012 INFO:tasks.rados.rados.0.vpm167.stdout:3363: writing vpm1675861-43 from 1818604 to 2292054 tid 2 2014-08-08T10:42:22.018 INFO:tasks.rados.rados.0.vpm167.stdout:3363: writing vpm1675861-43 from 2718166 to 2718167 tid 3 2014-08-08T10:42:22.019 INFO:tasks.rados.rados.0.vpm167.stdout:3364: rollback oid 37 current snap is 440 2014-08-08T10:42:22.019 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 37 to 432 2014-08-08T10:42:22.019 INFO:tasks.rados.rados.0.vpm167.stdout:3365: delete oid 16 current snap is 440 2014-08-08T10:42:22.040 INFO:tasks.rados.rados.0.vpm167.stdout:3360: expect (ObjNum 1197 snap 419 seq_num 1197) 2014-08-08T10:42:22.436 INFO:tasks.rados.rados.0.vpm167.stdout:3360: done (7 left) 2014-08-08T10:42:22.437 INFO:tasks.rados.rados.0.vpm167.stdout:3365: done (6 left) 2014-08-08T10:42:22.437 INFO:tasks.rados.rados.0.vpm167.stdout:3366: write oid 9 current snap is 440 2014-08-08T10:42:22.437 INFO:tasks.rados.rados.0.vpm167.stdout:3366: seq_num 1266 ranges {205778=1} 2014-08-08T10:42:22.438 INFO:tasks.rados.rados.0.vpm167.stdout:3366: writing vpm1675861-9 from 205778 to 205779 tid 1 2014-08-08T10:42:22.441 INFO:tasks.rados.rados.0.vpm167.stdout:3367: write oid 36 current snap is 440 2014-08-08T10:42:22.441 INFO:tasks.rados.rados.0.vpm167.stdout:3367: seq_num 1267 ranges {418384=1} 2014-08-08T10:42:22.442 INFO:tasks.rados.rados.0.vpm167.stdout:3367: writing vpm1675861-36 from 418384 to 418385 tid 1 2014-08-08T10:42:22.443 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 37 v 0 (ObjNum 81 snap 0 seq_num 46278576) dirty dne 2014-08-08T10:42:22.443 INFO:tasks.rados.rados.0.vpm167.stdout:3359: finishing write tid 1 to vpm1675861-41 2014-08-08T10:42:22.444 INFO:tasks.rados.rados.0.vpm167.stdout:3359: finishing write tid 3 to vpm1675861-41 2014-08-08T10:42:22.444 INFO:tasks.rados.rados.0.vpm167.stdout:3363: finishing write tid 1 to vpm1675861-43 2014-08-08T10:42:22.444 INFO:tasks.rados.rados.0.vpm167.stdout:3362: finishing write tid 1 to vpm1675861-17 2014-08-08T10:42:22.444 INFO:tasks.rados.rados.0.vpm167.stdout:3363: finishing write tid 2 to vpm1675861-43 2014-08-08T10:42:22.444 INFO:tasks.rados.rados.0.vpm167.stdout:3358: finishing write tid 1 to vpm1675861-45 2014-08-08T10:42:22.445 INFO:tasks.rados.rados.0.vpm167.stdout:3357: finishing write tid 1 to vpm1675861-4 2014-08-08T10:42:22.445 INFO:tasks.rados.rados.0.vpm167.stdout:3364: done (7 left) 2014-08-08T10:42:22.445 INFO:tasks.rados.rados.0.vpm167.stdout:3368: read oid 28 snap -1 2014-08-08T10:42:22.445 INFO:tasks.rados.rados.0.vpm167.stdout:3369: write oid 29 current snap is 440 2014-08-08T10:42:22.445 INFO:tasks.rados.rados.0.vpm167.stdout:3369: seq_num 1268 ranges {113613=1} 2014-08-08T10:42:22.448 INFO:tasks.rados.rados.0.vpm167.stdout:3369: writing vpm1675861-29 from 113613 to 113614 tid 1 2014-08-08T10:42:22.448 INFO:tasks.rados.rados.0.vpm167.stdout:3370: read oid 40 snap -1 2014-08-08T10:42:22.448 INFO:tasks.rados.rados.0.vpm167.stdout:3371: write oid 48 current snap is 440 2014-08-08T10:42:22.448 INFO:tasks.rados.rados.0.vpm167.stdout:3371: seq_num 1269 ranges {458444=400000} 2014-08-08T10:42:22.463 INFO:tasks.rados.rados.0.vpm167.stdout:3371: writing vpm1675861-48 from 458444 to 858444 tid 1 2014-08-08T10:42:22.465 INFO:tasks.rados.rados.0.vpm167.stdout:3370: done (10 left) 2014-08-08T10:42:22.466 INFO:tasks.rados.rados.0.vpm167.stdout:3372: write oid 44 current snap is 440 2014-08-08T10:42:22.466 INFO:tasks.rados.rados.0.vpm167.stdout:3372: seq_num 1270 ranges {403274=583995,1603273=1} 2014-08-08T10:42:22.489 INFO:tasks.rados.rados.0.vpm167.stdout:3372: writing vpm1675861-44 from 403274 to 987269 tid 1 2014-08-08T10:42:22.496 INFO:tasks.rados.rados.0.vpm167.stdout:3372: writing vpm1675861-44 from 1603273 to 1603274 tid 2 2014-08-08T10:42:22.497 INFO:tasks.rados.rados.0.vpm167.stdout:3368: expect (ObjNum 1252 snap 438 seq_num 1252) 2014-08-08T10:42:22.534 INFO:tasks.rados.rados.0.vpm167.stdout:3368: done (10 left) 2014-08-08T10:42:22.534 INFO:tasks.rados.rados.0.vpm167.stdout:3373: delete oid 42 current snap is 440 2014-08-08T10:42:22.535 INFO:tasks.rados.rados.0.vpm167.stdout:3366: finishing write tid 1 to vpm1675861-9 2014-08-08T10:42:22.599 INFO:tasks.rados.rados.0.vpm167.stdout:3367: finishing write tid 1 to vpm1675861-36 2014-08-08T10:42:22.606 INFO:tasks.rados.rados.0.vpm167.stdout:3362: finishing write tid 2 to vpm1675861-17 2014-08-08T10:42:22.607 INFO:tasks.rados.rados.0.vpm167.stdout:3369: finishing write tid 1 to vpm1675861-29 2014-08-08T10:42:22.611 INFO:tasks.rados.rados.0.vpm167.stdout:3366: finishing write tid 3 to vpm1675861-9 2014-08-08T10:42:22.612 INFO:tasks.rados.rados.0.vpm167.stdout:3373: done (10 left) 2014-08-08T10:42:22.613 INFO:tasks.rados.rados.0.vpm167.stdout:3374: write oid 7 current snap is 440 2014-08-08T10:42:22.613 INFO:tasks.rados.rados.0.vpm167.stdout:3374: seq_num 1272 ranges {559685=563700,1920526=571908,2559684=1} 2014-08-08T10:42:22.640 INFO:tasks.rados.rados.0.vpm167.stdout:3374: writing vpm1675861-7 from 559685 to 1123385 tid 1 2014-08-08T10:42:22.668 INFO:tasks.rados.rados.0.vpm167.stdout:3374: writing vpm1675861-7 from 1920526 to 2492434 tid 2 2014-08-08T10:42:22.670 INFO:tasks.rados.rados.0.vpm167.stdout:3374: writing vpm1675861-7 from 2559684 to 2559685 tid 3 2014-08-08T10:42:22.671 INFO:tasks.rados.rados.0.vpm167.stdout:3369: finishing write tid 3 to vpm1675861-29 2014-08-08T10:42:22.672 INFO:tasks.rados.rados.0.vpm167.stdout:3362: finishing write tid 4 to vpm1675861-17 2014-08-08T10:42:22.672 INFO:tasks.rados.rados.0.vpm167.stdout:3367: finishing write tid 3 to vpm1675861-36 2014-08-08T10:42:22.672 INFO:tasks.rados.rados.0.vpm167.stdout:3363: finishing write tid 3 to vpm1675861-43 2014-08-08T10:42:22.672 INFO:tasks.rados.rados.0.vpm167.stdout:3363: finishing write tid 5 to vpm1675861-43 2014-08-08T10:42:22.673 INFO:tasks.rados.rados.0.vpm167.stdout:3358: finishing write tid 3 to vpm1675861-45 2014-08-08T10:42:22.673 INFO:tasks.rados.rados.0.vpm167.stdout:3372: finishing write tid 1 to vpm1675861-44 2014-08-08T10:42:22.673 INFO:tasks.rados.rados.0.vpm167.stdout:3372: finishing write tid 2 to vpm1675861-44 2014-08-08T10:42:22.674 INFO:tasks.rados.rados.0.vpm167.stdout:3372: finishing write tid 4 to vpm1675861-44 2014-08-08T10:42:22.677 INFO:tasks.rados.rados.0.vpm167.stdout:3357: finishing write tid 3 to vpm1675861-4 2014-08-08T10:42:22.678 INFO:tasks.rados.rados.0.vpm167.stdout:3362: finishing write tid 5 to vpm1675861-17 2014-08-08T10:42:22.678 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2214 (ObjNum 1263 snap 440 seq_num 1263) dirty exists 2014-08-08T10:42:22.678 INFO:tasks.rados.rados.0.vpm167.stdout:3371: finishing write tid 1 to vpm1675861-48 2014-08-08T10:42:22.679 INFO:tasks.rados.rados.0.vpm167.stdout:3371: finishing write tid 3 to vpm1675861-48 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:3358: finishing write tid 4 to vpm1675861-45 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 1472 (ObjNum 1261 snap 440 seq_num 1261) dirty exists 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:3369: finishing write tid 4 to vpm1675861-29 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 1156 (ObjNum 1268 snap 440 seq_num 1268) dirty exists 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:3363: finishing write tid 6 to vpm1675861-43 2014-08-08T10:42:22.680 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 842 (ObjNum 1264 snap 440 seq_num 1264) dirty exists 2014-08-08T10:42:22.681 INFO:tasks.rados.rados.0.vpm167.stdout:3358: done (10 left) 2014-08-08T10:42:22.681 INFO:tasks.rados.rados.0.vpm167.stdout:3362: done (9 left) 2014-08-08T10:42:22.681 INFO:tasks.rados.rados.0.vpm167.stdout:3363: done (8 left) 2014-08-08T10:42:22.681 INFO:tasks.rados.rados.0.vpm167.stdout:3369: done (7 left) 2014-08-08T10:42:22.682 INFO:tasks.rados.rados.0.vpm167.stdout:3375: snap_remove snap 379 2014-08-08T10:42:22.682 INFO:tasks.rados.rados.0.vpm167.stdout:3374: finishing write tid 1 to vpm1675861-7 2014-08-08T10:42:22.942 INFO:tasks.rados.rados.0.vpm167.stdout:3374: finishing write tid 2 to vpm1675861-7 2014-08-08T10:42:24.387 INFO:tasks.rados.rados.0.vpm167.stdout:3375: done (7 left) 2014-08-08T10:42:24.388 INFO:tasks.rados.rados.0.vpm167.stdout:3376: write oid 49 current snap is 440 2014-08-08T10:42:24.388 INFO:tasks.rados.rados.0.vpm167.stdout:3376: seq_num 1273 ranges {502467=686928,1597538=642103,2975027=327440} 2014-08-08T10:42:24.413 INFO:tasks.rados.rados.0.vpm167.stdout:3376: writing vpm1675861-49 from 502467 to 1189395 tid 1 2014-08-08T10:42:24.443 INFO:tasks.rados.rados.0.vpm167.stdout:3376: writing vpm1675861-49 from 1597538 to 2239641 tid 2 2014-08-08T10:42:24.461 INFO:tasks.rados.rados.0.vpm167.stdout:3376: writing vpm1675861-49 from 2975027 to 3302467 tid 3 2014-08-08T10:42:24.463 INFO:tasks.rados.rados.0.vpm167.stdout:3359: finishing write tid 4 to vpm1675861-41 2014-08-08T10:42:24.463 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1793 (ObjNum 1262 snap 440 seq_num 1262) dirty exists 2014-08-08T10:42:24.463 INFO:tasks.rados.rados.0.vpm167.stdout:3374: finishing write tid 3 to vpm1675861-7 2014-08-08T10:42:24.464 INFO:tasks.rados.rados.0.vpm167.stdout:3374: finishing write tid 5 to vpm1675861-7 2014-08-08T10:42:24.464 INFO:tasks.rados.rados.0.vpm167.stdout:3366: finishing write tid 4 to vpm1675861-9 2014-08-08T10:42:24.464 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 9 v 1796 (ObjNum 1266 snap 440 seq_num 1266) dirty exists 2014-08-08T10:42:24.465 INFO:tasks.rados.rados.0.vpm167.stdout:3359: done (7 left) 2014-08-08T10:42:24.465 INFO:tasks.rados.rados.0.vpm167.stdout:3366: done (6 left) 2014-08-08T10:42:24.465 INFO:tasks.rados.rados.0.vpm167.stdout:3377: read oid 20 snap -1 2014-08-08T10:42:24.465 INFO:tasks.rados.rados.0.vpm167.stdout:3378: snap_remove snap 427 2014-08-08T10:42:24.490 INFO:tasks.rados.rados.0.vpm167.stdout:3377: expect (ObjNum 1041 snap 358 seq_num 1041) 2014-08-08T10:42:24.812 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:24.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:24.812 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-08T10:42:24.812 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:42:24.812 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:42:24.813 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-08T10:42:25.197 INFO:tasks.rados.rados.0.vpm167.stdout:3371: finishing write tid 4 to vpm1675861-48 2014-08-08T10:42:25.198 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 48 v 1810 (ObjNum 1269 snap 440 seq_num 1269) dirty exists 2014-08-08T10:42:25.198 INFO:tasks.rados.rados.0.vpm167.stdout:3376: finishing write tid 1 to vpm1675861-49 2014-08-08T10:42:25.198 INFO:tasks.rados.rados.0.vpm167.stdout:3376: finishing write tid 2 to vpm1675861-49 2014-08-08T10:42:25.198 INFO:tasks.rados.rados.0.vpm167.stdout:3376: finishing write tid 3 to vpm1675861-49 2014-08-08T10:42:25.198 INFO:tasks.rados.rados.0.vpm167.stdout:3357: finishing write tid 4 to vpm1675861-4 2014-08-08T10:42:25.199 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 4 v 1205 (ObjNum 1260 snap 440 seq_num 1260) dirty exists 2014-08-08T10:42:25.200 INFO:tasks.rados.rados.0.vpm167.stdout:3376: finishing write tid 5 to vpm1675861-49 2014-08-08T10:42:25.200 INFO:tasks.rados.rados.0.vpm167.stdout:3367: finishing write tid 4 to vpm1675861-36 2014-08-08T10:42:25.200 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 36 v 2126 (ObjNum 1267 snap 440 seq_num 1267) dirty exists 2014-08-08T10:42:25.200 INFO:tasks.rados.rados.0.vpm167.stdout:3372: finishing write tid 5 to vpm1675861-44 2014-08-08T10:42:25.201 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1478 (ObjNum 1270 snap 440 seq_num 1270) dirty exists 2014-08-08T10:42:25.201 INFO:tasks.rados.rados.0.vpm167.stdout:3374: finishing write tid 6 to vpm1675861-7 2014-08-08T10:42:25.201 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 837 (ObjNum 1272 snap 440 seq_num 1272) dirty exists 2014-08-08T10:42:25.201 INFO:tasks.rados.rados.0.vpm167.stdout:3376: finishing write tid 6 to vpm1675861-49 2014-08-08T10:42:25.201 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 49 v 1345 (ObjNum 1273 snap 440 seq_num 1273) dirty exists 2014-08-08T10:42:25.545 INFO:tasks.rados.rados.0.vpm167.stdout:3357: done (7 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3367: done (6 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3371: done (5 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3372: done (4 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3374: done (3 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3376: done (2 left) 2014-08-08T10:42:25.546 INFO:tasks.rados.rados.0.vpm167.stdout:3377: done (1 left) 2014-08-08T10:42:25.547 INFO:tasks.rados.rados.0.vpm167.stdout:3378: done (0 left) 2014-08-08T10:42:25.547 INFO:tasks.rados.rados.0.vpm167.stdout:3379: write oid 41 current snap is 440 2014-08-08T10:42:25.547 INFO:tasks.rados.rados.0.vpm167.stdout:3379: seq_num 1274 ranges {598721=659756,1712694=463722,2778300=220421} 2014-08-08T10:42:25.580 INFO:tasks.rados.rados.0.vpm167.stdout:3379: writing vpm1675861-41 from 598721 to 1258477 tid 1 2014-08-08T10:42:25.615 INFO:tasks.rados.rados.0.vpm167.stdout:3379: writing vpm1675861-41 from 1712694 to 2176416 tid 2 2014-08-08T10:42:25.630 INFO:tasks.rados.rados.0.vpm167.stdout:3379: writing vpm1675861-41 from 2778300 to 2998721 tid 3 2014-08-08T10:42:25.631 INFO:tasks.rados.rados.0.vpm167.stdout:3380: rollback oid 5 current snap is 440 2014-08-08T10:42:25.631 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 5 to 344 2014-08-08T10:42:25.632 INFO:tasks.rados.rados.0.vpm167.stdout:3381: delete oid 29 current snap is 440 2014-08-08T10:42:25.639 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 5 v 1215 (ObjNum 1005 snap 342 seq_num 1005) dirty exists 2014-08-08T10:42:25.676 INFO:tasks.rados.rados.0.vpm167.stdout:3380: done (2 left) 2014-08-08T10:42:25.676 INFO:tasks.rados.rados.0.vpm167.stdout:3381: done (1 left) 2014-08-08T10:42:25.676 INFO:tasks.rados.rados.0.vpm167.stdout:3382: write oid 43 current snap is 440 2014-08-08T10:42:25.677 INFO:tasks.rados.rados.0.vpm167.stdout:3382: seq_num 1276 ranges {603708=400000} 2014-08-08T10:42:25.693 INFO:tasks.rados.rados.0.vpm167.stdout:3382: writing vpm1675861-43 from 603708 to 1003708 tid 1 2014-08-08T10:42:25.698 INFO:tasks.rados.rados.0.vpm167.stdout:3383: rollback oid 25 current snap is 440 2014-08-08T10:42:25.698 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 25 to 439 2014-08-08T10:42:25.699 INFO:tasks.rados.rados.0.vpm167.stdout:3384: read oid 45 snap -1 2014-08-08T10:42:25.699 INFO:tasks.rados.rados.0.vpm167.stdout:3385: delete oid 36 current snap is 440 2014-08-08T10:42:25.716 INFO:tasks.rados.rados.0.vpm167.stdout:3382: finishing write tid 1 to vpm1675861-43 2014-08-08T10:42:25.722 INFO:tasks.rados.rados.0.vpm167.stdout:3382: finishing write tid 3 to vpm1675861-43 2014-08-08T10:42:25.729 INFO:tasks.rados.rados.0.vpm167.stdout:3384: expect (ObjNum 1261 snap 440 seq_num 1261) 2014-08-08T10:42:25.915 INFO:tasks.rados.rados.0.vpm167.stdout:3379: finishing write tid 1 to vpm1675861-41 2014-08-08T10:42:25.941 INFO:tasks.rados.rados.0.vpm167.stdout:3379: finishing write tid 2 to vpm1675861-41 2014-08-08T10:42:25.947 INFO:tasks.rados.rados.0.vpm167.stdout:3379: finishing write tid 3 to vpm1675861-41 2014-08-08T10:42:25.974 INFO:tasks.rados.rados.0.vpm167.stdout:3379: finishing write tid 5 to vpm1675861-41 2014-08-08T10:42:25.981 INFO:tasks.rados.rados.0.vpm167.stdout:3379: finishing write tid 6 to vpm1675861-41 2014-08-08T10:42:25.981 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1806 (ObjNum 1274 snap 440 seq_num 1274) dirty exists 2014-08-08T10:42:26.215 INFO:tasks.rados.rados.0.vpm167.stdout:3379: done (4 left) 2014-08-08T10:42:26.216 INFO:tasks.rados.rados.0.vpm167.stdout:3384: done (3 left) 2014-08-08T10:42:26.217 INFO:tasks.rados.rados.0.vpm167.stdout:3385: done (2 left) 2014-08-08T10:42:26.217 INFO:tasks.rados.rados.0.vpm167.stdout:3386: rollback oid 24 current snap is 440 2014-08-08T10:42:26.217 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 24 to 438 2014-08-08T10:42:26.217 INFO:tasks.rados.rados.0.vpm167.stdout:3387: read oid 32 snap -1 2014-08-08T10:42:26.218 INFO:tasks.rados.rados.0.vpm167.stdout:3388: read oid 7 snap 419 2014-08-08T10:42:26.219 INFO:tasks.rados.rados.0.vpm167.stdout:3389: rollback oid 14 current snap is 440 2014-08-08T10:42:26.219 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 415 2014-08-08T10:42:26.219 INFO:tasks.rados.rados.0.vpm167.stdout:3390: write oid 44 current snap is 440 2014-08-08T10:42:26.219 INFO:tasks.rados.rados.0.vpm167.stdout:3390: seq_num 1278 ranges {643768=734494,1443767=1} 2014-08-08T10:42:26.256 INFO:tasks.rados.rados.0.vpm167.stdout:3390: writing vpm1675861-44 from 643768 to 1378262 tid 1 2014-08-08T10:42:26.263 INFO:tasks.rados.rados.0.vpm167.stdout:3390: writing vpm1675861-44 from 1443767 to 1443768 tid 2 2014-08-08T10:42:26.269 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 25 v 871 (ObjNum 1048 snap 360 seq_num 1048) dirty exists 2014-08-08T10:42:26.270 INFO:tasks.rados.rados.0.vpm167.stdout:3387: expect (ObjNum 1212 snap 423 seq_num 1212) 2014-08-08T10:42:26.333 INFO:tasks.rados.rados.0.vpm167.stdout:3383: done (6 left) 2014-08-08T10:42:26.333 INFO:tasks.rados.rados.0.vpm167.stdout:3387: done (5 left) 2014-08-08T10:42:26.334 INFO:tasks.rados.rados.0.vpm167.stdout:3391: snap_remove snap 439 2014-08-08T10:42:26.334 INFO:tasks.rados.rados.0.vpm167.stdout:3388: expect (ObjNum 1083 snap 376 seq_num 1083) 2014-08-08T10:42:26.758 INFO:tasks.rados.rados.0.vpm167.stdout:3388: done (5 left) 2014-08-08T10:42:26.759 INFO:tasks.rados.rados.0.vpm167.stdout:3391: done (4 left) 2014-08-08T10:42:26.759 INFO:tasks.rados.rados.0.vpm167.stdout:3392: snap_create 2014-08-08T10:42:26.760 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2143 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:42:26.761 INFO:tasks.rados.rados.0.vpm167.stdout:3390: finishing write tid 1 to vpm1675861-44 2014-08-08T10:42:26.761 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 24 v 0 (ObjNum 320 snap 0 seq_num 4294967295) dirty dne 2014-08-08T10:42:26.761 INFO:tasks.rados.rados.0.vpm167.stdout:3390: finishing write tid 2 to vpm1675861-44 2014-08-08T10:42:26.761 INFO:tasks.rados.rados.0.vpm167.stdout:3390: finishing write tid 4 to vpm1675861-44 2014-08-08T10:42:26.762 INFO:tasks.rados.rados.0.vpm167.stdout:3390: finishing write tid 5 to vpm1675861-44 2014-08-08T10:42:26.762 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1487 (ObjNum 1278 snap 440 seq_num 1278) dirty exists 2014-08-08T10:42:27.785 INFO:tasks.rados.rados.0.vpm167.stdout:3386: done (4 left) 2014-08-08T10:42:27.785 INFO:tasks.rados.rados.0.vpm167.stdout:3389: done (3 left) 2014-08-08T10:42:27.786 INFO:tasks.rados.rados.0.vpm167.stdout:3390: done (2 left) 2014-08-08T10:42:27.786 INFO:tasks.rados.rados.0.vpm167.stdout:3392: done (1 left) 2014-08-08T10:42:27.786 INFO:tasks.rados.rados.0.vpm167.stdout:3393: read oid 18 snap -1 2014-08-08T10:42:27.786 INFO:tasks.rados.rados.0.vpm167.stdout:3394: write oid 7 current snap is 441 2014-08-08T10:42:27.787 INFO:tasks.rados.rados.0.vpm167.stdout:3394: seq_num 1279 ranges {588597=458746,1660762=527835} 2014-08-08T10:42:27.805 INFO:tasks.rados.rados.0.vpm167.stdout:3394: writing vpm1675861-7 from 588597 to 1047343 tid 1 2014-08-08T10:42:27.835 INFO:tasks.rados.rados.0.vpm167.stdout:3394: writing vpm1675861-7 from 1660762 to 2188597 tid 2 2014-08-08T10:42:27.837 INFO:tasks.rados.rados.0.vpm167.stdout:3395: read oid 29 snap 440 2014-08-08T10:42:27.837 INFO:tasks.rados.rados.0.vpm167.stdout:3396: snap_remove snap 410 2014-08-08T10:42:27.858 INFO:tasks.rados.rados.0.vpm167.stdout:3394: finishing write tid 1 to vpm1675861-7 2014-08-08T10:42:27.870 INFO:tasks.rados.rados.0.vpm167.stdout:3394: finishing write tid 2 to vpm1675861-7 2014-08-08T10:42:27.870 INFO:tasks.rados.rados.0.vpm167.stdout:3394: finishing write tid 4 to vpm1675861-7 2014-08-08T10:42:27.908 INFO:tasks.rados.rados.0.vpm167.stdout:3393: expect (ObjNum 1234 snap 432 seq_num 1234) 2014-08-08T10:42:28.222 INFO:tasks.rados.rados.0.vpm167.stdout:3382: finishing write tid 4 to vpm1675861-43 2014-08-08T10:42:28.222 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 846 (ObjNum 1276 snap 440 seq_num 1276) dirty exists 2014-08-08T10:42:28.336 INFO:tasks.rados.rados.0.vpm167.stdout:3394: finishing write tid 5 to vpm1675861-7 2014-08-08T10:42:28.336 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 7 v 849 (ObjNum 1279 snap 441 seq_num 1279) dirty exists 2014-08-08T10:42:28.836 INFO:tasks.rados.rados.0.vpm167.stdout:3382: done (4 left) 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3393: done (3 left) 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3394: done (2 left) 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3395: done (1 left) 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3396: done (0 left) 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3397: read oid 13 snap -1 2014-08-08T10:42:28.837 INFO:tasks.rados.rados.0.vpm167.stdout:3398: snap_remove snap 428 2014-08-08T10:42:29.083 INFO:tasks.rados.rados.0.vpm167.stdout:3397: expect (ObjNum 1250 snap 438 seq_num 1250) 2014-08-08T10:42:30.001 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:30.002 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T10:42:31.410 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T10:42:31.448 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-08T10:42:31.838 INFO:tasks.rados.rados.0.vpm167.stdout:3397: done (1 left) 2014-08-08T10:42:31.839 INFO:tasks.rados.rados.0.vpm167.stdout:3398: done (0 left) 2014-08-08T10:42:31.839 INFO:tasks.rados.rados.0.vpm167.stdout:3399: snap_remove snap 433 2014-08-08T10:42:32.638 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T10:42:32.642 INFO:tasks.rados.rados.0.vpm167.stdout:3399: done (0 left) 2014-08-08T10:42:32.643 INFO:tasks.rados.rados.0.vpm167.stdout:3400: read oid 32 snap -1 2014-08-08T10:42:32.643 INFO:tasks.rados.rados.0.vpm167.stdout:3401: snap_remove snap 416 2014-08-08T10:42:32.659 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T10:42:32.660 INFO:tasks.rados.rados.0.vpm167.stdout:3400: expect (ObjNum 1212 snap 423 seq_num 1212) 2014-08-08T10:42:35.046 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T10:42:35.065 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T10:42:35.909 INFO:tasks.rados.rados.0.vpm167.stdout:3400: done (1 left) 2014-08-08T10:42:35.909 INFO:tasks.rados.rados.0.vpm167.stdout:3401: done (0 left) 2014-08-08T10:42:35.909 INFO:tasks.rados.rados.0.vpm167.stdout:3402: snap_create 2014-08-08T10:42:36.908 INFO:tasks.rados.rados.0.vpm167.stdout:3402: done (0 left) 2014-08-08T10:42:36.909 INFO:tasks.rados.rados.0.vpm167.stdout:3403: write oid 39 current snap is 442 2014-08-08T10:42:36.909 INFO:tasks.rados.rados.0.vpm167.stdout:3403: seq_num 1280 ranges {684851=714198,1884850=1} 2014-08-08T10:42:36.939 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T10:42:36.939 INFO:tasks.rados.rados.0.vpm167.stdout:3403: writing vpm1675861-39 from 684851 to 1399049 tid 1 2014-08-08T10:42:36.953 INFO:tasks.rados.rados.0.vpm167.stdout:3403: writing vpm1675861-39 from 1884850 to 1884851 tid 2 2014-08-08T10:42:36.953 INFO:tasks.rados.rados.0.vpm167.stdout:3404: snap_remove snap 438 2014-08-08T10:42:36.967 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T10:42:37.190 INFO:tasks.rados.rados.0.vpm167.stdout:3403: finishing write tid 1 to vpm1675861-39 2014-08-08T10:42:37.190 INFO:tasks.rados.rados.0.vpm167.stdout:3403: finishing write tid 2 to vpm1675861-39 2014-08-08T10:42:37.191 INFO:tasks.rados.rados.0.vpm167.stdout:3403: finishing write tid 4 to vpm1675861-39 2014-08-08T10:42:37.202 INFO:tasks.rados.rados.0.vpm167.stdout:3403: finishing write tid 5 to vpm1675861-39 2014-08-08T10:42:37.202 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 39 v 1505 (ObjNum 1280 snap 442 seq_num 1280) dirty exists 2014-08-08T10:42:37.986 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T10:42:38.020 INFO:tasks.rados.rados.0.vpm167.stdout:3403: done (1 left) 2014-08-08T10:42:38.020 INFO:tasks.rados.rados.0.vpm167.stdout:3404: done (0 left) 2014-08-08T10:42:38.021 INFO:tasks.rados.rados.0.vpm167.stdout:3405: snap_create 2014-08-08T10:42:38.027 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T10:42:38.027 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:42:38.638 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:42:38.693 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:42:38.922 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:42:38.996 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:42:39.177 INFO:tasks.rados.rados.0.vpm167.stdout:3405: done (0 left) 2014-08-08T10:42:39.177 INFO:tasks.rados.rados.0.vpm167.stdout:3406: write oid 13 current snap is 443 2014-08-08T10:42:39.178 INFO:tasks.rados.rados.0.vpm167.stdout:3406: seq_num 1281 ranges {743985=687027,1852931=741584,3143984=1} 2014-08-08T10:42:39.205 INFO:tasks.rados.rados.0.vpm167.stdout:3406: writing vpm1675861-13 from 743985 to 1431012 tid 1 2014-08-08T10:42:39.242 INFO:tasks.rados.rados.0.vpm167.stdout:3406: writing vpm1675861-13 from 1852931 to 2594515 tid 2 2014-08-08T10:42:39.248 INFO:tasks.rados.rados.0.vpm167.stdout:3406: writing vpm1675861-13 from 3143984 to 3143985 tid 3 2014-08-08T10:42:39.249 INFO:tasks.rados.rados.0.vpm167.stdout:3407: delete oid 35 current snap is 443 2014-08-08T10:42:39.258 INFO:tasks.rados.rados.0.vpm167.stdout:3407: done (1 left) 2014-08-08T10:42:39.258 INFO:tasks.rados.rados.0.vpm167.stdout:3408: snap_create 2014-08-08T10:42:39.275 INFO:tasks.rados.rados.0.vpm167.stdout:3406: finishing write tid 1 to vpm1675861-13 2014-08-08T10:42:39.304 INFO:tasks.rados.rados.0.vpm167.stdout:3406: finishing write tid 2 to vpm1675861-13 2014-08-08T10:42:39.304 INFO:tasks.rados.rados.0.vpm167.stdout:3406: finishing write tid 3 to vpm1675861-13 2014-08-08T10:42:39.319 INFO:tasks.rados.rados.0.vpm167.stdout:3406: finishing write tid 5 to vpm1675861-13 2014-08-08T10:42:39.365 INFO:tasks.rados.rados.0.vpm167.stdout:3406: finishing write tid 6 to vpm1675861-13 2014-08-08T10:42:39.365 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1859 (ObjNum 1281 snap 443 seq_num 1281) dirty exists 2014-08-08T10:42:39.594 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:42:19.622502', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88449900, u'health': u'HEALTH_OK', u'kb_used': 10503668, u'store_stats': {u'bytes_total': 32456249, u'bytes_log': 5602837, u'last_updated': u'0.000000', u'bytes_misc': 26853412, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:26.393339', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87316340, u'health': u'HEALTH_OK', u'kb_used': 11637228, u'store_stats': {u'bytes_total': 36096972, u'bytes_log': 9257588, u'last_updated': u'0.000000', u'bytes_misc': 26839384, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:23.352345', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88446140, u'health': u'HEALTH_OK', u'kb_used': 10507428, u'store_stats': {u'bytes_total': 35693429, u'bytes_log': 8833927, u'last_updated': u'0.000000', u'bytes_misc': 26859502, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 54 > pgp_num 44'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'num_pgs': 348, u'data_bytes': 305371104, u'bytes_used': 1642852352, u'version': 2406, u'pgs_by_state': [{u'count': 348, u'state_name': u'active+clean'}], u'bytes_avail': 1071574683648}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1300, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:42:39.595 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T10:42:39.596 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:39.596 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-08T10:42:39.596 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:42:39.597 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:42:39.597 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-08T10:42:40.270 INFO:tasks.rados.rados.0.vpm167.stdout:3406: done (1 left) 2014-08-08T10:42:40.270 INFO:tasks.rados.rados.0.vpm167.stdout:3408: done (0 left) 2014-08-08T10:42:40.270 INFO:tasks.rados.rados.0.vpm167.stdout:3409: snap_create 2014-08-08T10:42:41.360 INFO:tasks.rados.rados.0.vpm167.stdout:3409: done (0 left) 2014-08-08T10:42:41.360 INFO:tasks.rados.rados.0.vpm167.stdout:3410: snap_remove snap 436 2014-08-08T10:42:42.448 INFO:tasks.rados.rados.0.vpm167.stdout:3410: done (0 left) 2014-08-08T10:42:42.448 INFO:tasks.rados.rados.0.vpm167.stdout:3411: read oid 30 snap -1 2014-08-08T10:42:42.449 INFO:tasks.rados.rados.0.vpm167.stdout:3412: read oid 14 snap -1 2014-08-08T10:42:42.449 INFO:tasks.rados.rados.0.vpm167.stdout:3413: read oid 19 snap -1 2014-08-08T10:42:42.450 INFO:tasks.rados.rados.0.vpm167.stdout:3414: read oid 43 snap 444 2014-08-08T10:42:42.451 INFO:tasks.rados.rados.0.vpm167.stdout:3415: write oid 3 current snap is 445 2014-08-08T10:42:42.451 INFO:tasks.rados.rados.0.vpm167.stdout:3415: seq_num 1283 ranges {784044=782060,2045100=633399,3335057=248987} 2014-08-08T10:42:42.481 INFO:tasks.rados.rados.0.vpm167.stdout:3415: writing vpm1675861-3 from 784044 to 1566104 tid 1 2014-08-08T10:42:42.519 INFO:tasks.rados.rados.0.vpm167.stdout:3415: writing vpm1675861-3 from 2045100 to 2678499 tid 2 2014-08-08T10:42:42.536 INFO:tasks.rados.rados.0.vpm167.stdout:3415: writing vpm1675861-3 from 3335057 to 3584044 tid 3 2014-08-08T10:42:42.541 INFO:tasks.rados.rados.0.vpm167.stdout:3414: expect (ObjNum 1276 snap 440 seq_num 1276) 2014-08-08T10:42:42.741 INFO:tasks.rados.rados.0.vpm167.stdout:3414: done (4 left) 2014-08-08T10:42:42.742 INFO:tasks.rados.rados.0.vpm167.stdout:3416: delete oid 29 current snap is 445 2014-08-08T10:42:42.742 INFO:tasks.rados.rados.0.vpm167.stdout:3413: expect (ObjNum 1197 snap 419 seq_num 1197) 2014-08-08T10:42:43.083 INFO:tasks.rados.rados.0.vpm167.stdout:3413: done (4 left) 2014-08-08T10:42:43.083 INFO:tasks.rados.rados.0.vpm167.stdout:3416: done (3 left) 2014-08-08T10:42:43.083 INFO:tasks.rados.rados.0.vpm167.stdout:3417: rollback oid 42 current snap is 445 2014-08-08T10:42:43.084 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 42 to 408 2014-08-08T10:42:43.084 INFO:tasks.rados.rados.0.vpm167.stdout:3418: delete oid 25 current snap is 445 2014-08-08T10:42:43.085 INFO:tasks.rados.rados.0.vpm167.stdout:3415: finishing write tid 1 to vpm1675861-3 2014-08-08T10:42:43.085 INFO:tasks.rados.rados.0.vpm167.stdout:3415: finishing write tid 2 to vpm1675861-3 2014-08-08T10:42:43.085 INFO:tasks.rados.rados.0.vpm167.stdout:3412: expect (ObjNum 1172 snap 412 seq_num 1172) 2014-08-08T10:42:43.259 INFO:tasks.rados.rados.0.vpm167.stdout:3412: done (4 left) 2014-08-08T10:42:43.259 INFO:tasks.rados.rados.0.vpm167.stdout:3418: done (3 left) 2014-08-08T10:42:43.260 INFO:tasks.rados.rados.0.vpm167.stdout:3419: read oid 16 snap -1 2014-08-08T10:42:43.260 INFO:tasks.rados.rados.0.vpm167.stdout:3420: snap_create 2014-08-08T10:42:43.260 INFO:tasks.rados.rados.0.vpm167.stdout:3415: finishing write tid 3 to vpm1675861-3 2014-08-08T10:42:43.260 INFO:tasks.rados.rados.0.vpm167.stdout:3415: finishing write tid 5 to vpm1675861-3 2014-08-08T10:42:43.261 INFO:tasks.rados.rados.0.vpm167.stdout:3411: expect (ObjNum 1052 snap 362 seq_num 1052) 2014-08-08T10:42:44.055 INFO:tasks.rados.rados.0.vpm167.stdout:3411: done (4 left) 2014-08-08T10:42:44.055 INFO:tasks.rados.rados.0.vpm167.stdout:3420: done (3 left) 2014-08-08T10:42:44.055 INFO:tasks.rados.rados.0.vpm167.stdout:3421: write oid 19 current snap is 446 2014-08-08T10:42:44.056 INFO:tasks.rados.rados.0.vpm167.stdout:3421: seq_num 1286 ranges {485286=733569,1285285=1} 2014-08-08T10:42:44.082 INFO:tasks.rados.rados.0.vpm167.stdout:3421: writing vpm1675861-19 from 485286 to 1218855 tid 1 2014-08-08T10:42:44.087 INFO:tasks.rados.rados.0.vpm167.stdout:3421: writing vpm1675861-19 from 1285285 to 1285286 tid 2 2014-08-08T10:42:44.088 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 0 (ObjNum -1017313344 snap 32533 seq_num 1703937) dirty dne 2014-08-08T10:42:44.089 INFO:tasks.rados.rados.0.vpm167.stdout:3415: finishing write tid 6 to vpm1675861-3 2014-08-08T10:42:44.089 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 866 (ObjNum 1283 snap 445 seq_num 1283) dirty exists 2014-08-08T10:42:44.089 INFO:tasks.rados.rados.0.vpm167.stdout:3415: done (3 left) 2014-08-08T10:42:44.090 INFO:tasks.rados.rados.0.vpm167.stdout:3417: done (2 left) 2014-08-08T10:42:44.090 INFO:tasks.rados.rados.0.vpm167.stdout:3419: done (1 left) 2014-08-08T10:42:44.090 INFO:tasks.rados.rados.0.vpm167.stdout:3422: write oid 30 current snap is 446 2014-08-08T10:42:44.090 INFO:tasks.rados.rados.0.vpm167.stdout:3422: seq_num 1287 ranges {430115=456797,1514108=516007} 2014-08-08T10:42:44.107 INFO:tasks.rados.rados.0.vpm167.stdout:3422: writing vpm1675861-30 from 430115 to 886912 tid 1 2014-08-08T10:42:44.132 INFO:tasks.rados.rados.0.vpm167.stdout:3422: writing vpm1675861-30 from 1514108 to 2030115 tid 2 2014-08-08T10:42:44.134 INFO:tasks.rados.rados.0.vpm167.stdout:3421: finishing write tid 1 to vpm1675861-19 2014-08-08T10:42:44.134 INFO:tasks.rados.rados.0.vpm167.stdout:3421: finishing write tid 2 to vpm1675861-19 2014-08-08T10:42:44.135 INFO:tasks.rados.rados.0.vpm167.stdout:3422: finishing write tid 1 to vpm1675861-30 2014-08-08T10:42:44.135 INFO:tasks.rados.rados.0.vpm167.stdout:3421: finishing write tid 4 to vpm1675861-19 2014-08-08T10:42:44.135 INFO:tasks.rados.rados.0.vpm167.stdout:3421: finishing write tid 5 to vpm1675861-19 2014-08-08T10:42:44.135 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 19 v 1402 (ObjNum 1286 snap 446 seq_num 1286) dirty exists 2014-08-08T10:42:44.136 INFO:tasks.rados.rados.0.vpm167.stdout:3421: done (1 left) 2014-08-08T10:42:44.136 INFO:tasks.rados.rados.0.vpm167.stdout:3423: write oid 3 current snap is 446 2014-08-08T10:42:44.136 INFO:tasks.rados.rados.0.vpm167.stdout:3423: seq_num 1288 ranges {525345=713273,1725344=1} 2014-08-08T10:42:44.163 INFO:tasks.rados.rados.0.vpm167.stdout:3423: writing vpm1675861-3 from 525345 to 1238618 tid 1 2014-08-08T10:42:44.168 INFO:tasks.rados.rados.0.vpm167.stdout:3423: writing vpm1675861-3 from 1725344 to 1725345 tid 2 2014-08-08T10:42:44.169 INFO:tasks.rados.rados.0.vpm167.stdout:3422: finishing write tid 2 to vpm1675861-30 2014-08-08T10:42:44.169 INFO:tasks.rados.rados.0.vpm167.stdout:3422: finishing write tid 4 to vpm1675861-30 2014-08-08T10:42:44.170 INFO:tasks.rados.rados.0.vpm167.stdout:3422: finishing write tid 5 to vpm1675861-30 2014-08-08T10:42:44.170 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 30 v 1251 (ObjNum 1287 snap 446 seq_num 1287) dirty exists 2014-08-08T10:42:44.170 INFO:tasks.rados.rados.0.vpm167.stdout:3422: done (1 left) 2014-08-08T10:42:44.171 INFO:tasks.rados.rados.0.vpm167.stdout:3424: rollback oid 24 current snap is 446 2014-08-08T10:42:44.171 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 24 to 443 2014-08-08T10:42:44.171 INFO:tasks.rados.rados.0.vpm167.stdout:3425: snap_create 2014-08-08T10:42:44.176 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 24 v 0 (ObjNum 30006688 snap 0 seq_num 0) dirty dne 2014-08-08T10:42:44.192 INFO:tasks.rados.rados.0.vpm167.stdout:3423: finishing write tid 1 to vpm1675861-3 2014-08-08T10:42:44.199 INFO:tasks.rados.rados.0.vpm167.stdout:3423: finishing write tid 2 to vpm1675861-3 2014-08-08T10:42:44.199 INFO:tasks.rados.rados.0.vpm167.stdout:3423: finishing write tid 4 to vpm1675861-3 2014-08-08T10:42:44.584 INFO:tasks.rados.rados.0.vpm167.stdout:3424: done (2 left) 2014-08-08T10:42:44.584 INFO:tasks.rados.rados.0.vpm167.stdout:3425: done (1 left) 2014-08-08T10:42:44.584 INFO:tasks.rados.rados.0.vpm167.stdout:3426: snap_create 2014-08-08T10:42:44.730 INFO:tasks.rados.rados.0.vpm167.stdout:3423: finishing write tid 5 to vpm1675861-3 2014-08-08T10:42:44.730 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 872 (ObjNum 1288 snap 446 seq_num 1288) dirty exists 2014-08-08T10:42:44.844 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:44.845 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:44.845 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T10:42:44.845 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:42:44.845 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:42:44.845 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T10:42:45.960 INFO:tasks.rados.rados.0.vpm167.stdout:3423: done (1 left) 2014-08-08T10:42:45.962 INFO:tasks.rados.rados.0.vpm167.stdout:3426: done (0 left) 2014-08-08T10:42:45.962 INFO:tasks.rados.rados.0.vpm167.stdout:3427: write oid 35 current snap is 448 2014-08-08T10:42:45.962 INFO:tasks.rados.rados.0.vpm167.stdout:3427: seq_num 1289 ranges {469151=436502,1589925=441066,2469150=1} 2014-08-08T10:42:45.982 INFO:tasks.rados.rados.0.vpm167.stdout:3427: writing vpm1675861-35 from 469151 to 905653 tid 1 2014-08-08T10:42:46.007 INFO:tasks.rados.rados.0.vpm167.stdout:3427: writing vpm1675861-35 from 1589925 to 2030991 tid 2 2014-08-08T10:42:46.013 INFO:tasks.rados.rados.0.vpm167.stdout:3427: writing vpm1675861-35 from 2469150 to 2469151 tid 3 2014-08-08T10:42:46.014 INFO:tasks.rados.rados.0.vpm167.stdout:3428: write oid 26 current snap is 448 2014-08-08T10:42:46.015 INFO:tasks.rados.rados.0.vpm167.stdout:3428: seq_num 1290 ranges {530333=408306,1385337=662685,2574035=615643,3730332=1} 2014-08-08T10:42:46.041 INFO:tasks.rados.rados.0.vpm167.stdout:3428: writing vpm1675861-26 from 530333 to 938639 tid 1 2014-08-08T10:42:46.085 INFO:tasks.rados.rados.0.vpm167.stdout:3428: writing vpm1675861-26 from 1385337 to 2048022 tid 2 2014-08-08T10:42:46.123 INFO:tasks.rados.rados.0.vpm167.stdout:3428: writing vpm1675861-26 from 2574035 to 3189678 tid 3 2014-08-08T10:42:46.131 INFO:tasks.rados.rados.0.vpm167.stdout:3428: writing vpm1675861-26 from 3730332 to 3730333 tid 4 2014-08-08T10:42:46.132 INFO:tasks.rados.rados.0.vpm167.stdout:3427: finishing write tid 1 to vpm1675861-35 2014-08-08T10:42:46.132 INFO:tasks.rados.rados.0.vpm167.stdout:3429: read oid 34 snap 408 2014-08-08T10:42:46.133 INFO:tasks.rados.rados.0.vpm167.stdout:3430: rollback oid 22 current snap is 448 2014-08-08T10:42:46.133 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 22 to 384 2014-08-08T10:42:46.133 INFO:tasks.rados.rados.0.vpm167.stdout:3431: snap_remove snap 413 2014-08-08T10:42:46.140 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 22 v 0 (ObjNum -1043019704 snap 32533 seq_num 0) dirty dne 2014-08-08T10:42:46.937 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 1 to vpm1675861-26 2014-08-08T10:42:46.946 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 2 to vpm1675861-26 2014-08-08T10:42:47.065 INFO:tasks.rados.rados.0.vpm167.stdout:3429: done (4 left) 2014-08-08T10:42:47.065 INFO:tasks.rados.rados.0.vpm167.stdout:3430: done (3 left) 2014-08-08T10:42:47.065 INFO:tasks.rados.rados.0.vpm167.stdout:3431: done (2 left) 2014-08-08T10:42:47.065 INFO:tasks.rados.rados.0.vpm167.stdout:3432: rollback oid 18 current snap is 448 2014-08-08T10:42:47.066 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 18 to 440 2014-08-08T10:42:47.066 INFO:tasks.rados.rados.0.vpm167.stdout:3433: read oid 21 snap -1 2014-08-08T10:42:47.067 INFO:tasks.rados.rados.0.vpm167.stdout:3434: snap_create 2014-08-08T10:42:47.269 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 3 to vpm1675861-26 2014-08-08T10:42:47.270 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 4 to vpm1675861-26 2014-08-08T10:42:47.271 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 6 to vpm1675861-26 2014-08-08T10:42:47.280 INFO:tasks.rados.rados.0.vpm167.stdout:3427: finishing write tid 2 to vpm1675861-35 2014-08-08T10:42:47.281 INFO:tasks.rados.rados.0.vpm167.stdout:3427: finishing write tid 3 to vpm1675861-35 2014-08-08T10:42:47.285 INFO:tasks.rados.rados.0.vpm167.stdout:3427: finishing write tid 5 to vpm1675861-35 2014-08-08T10:42:47.285 INFO:tasks.rados.rados.0.vpm167.stdout:3427: finishing write tid 6 to vpm1675861-35 2014-08-08T10:42:47.285 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 35 v 1197 (ObjNum 1289 snap 448 seq_num 1289) dirty exists 2014-08-08T10:42:47.312 INFO:tasks.rados.rados.0.vpm167.stdout:3433: expect (ObjNum 1010 snap 345 seq_num 1010) 2014-08-08T10:42:47.947 INFO:tasks.rados.rados.0.vpm167.stdout:3428: finishing write tid 7 to vpm1675861-26 2014-08-08T10:42:47.947 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 953 (ObjNum 1290 snap 448 seq_num 1290) dirty exists 2014-08-08T10:42:48.731 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 18 v 2175 (ObjNum 1234 snap 432 seq_num 1234) dirty exists 2014-08-08T10:42:48.971 INFO:tasks.rados.rados.0.vpm167.stdout:3427: done (4 left) 2014-08-08T10:42:48.972 INFO:tasks.rados.rados.0.vpm167.stdout:3428: done (3 left) 2014-08-08T10:42:48.972 INFO:tasks.rados.rados.0.vpm167.stdout:3432: done (2 left) 2014-08-08T10:42:48.972 INFO:tasks.rados.rados.0.vpm167.stdout:3433: done (1 left) 2014-08-08T10:42:48.973 INFO:tasks.rados.rados.0.vpm167.stdout:3434: done (0 left) 2014-08-08T10:42:48.973 INFO:tasks.rados.rados.0.vpm167.stdout:3435: delete oid 6 current snap is 449 2014-08-08T10:42:49.890 INFO:tasks.rados.rados.0.vpm167.stdout:3435: done (0 left) 2014-08-08T10:42:49.890 INFO:tasks.rados.rados.0.vpm167.stdout:3436: rollback oid 50 current snap is 449 2014-08-08T10:42:49.890 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 50 to 447 2014-08-08T10:42:49.890 INFO:tasks.rados.rados.0.vpm167.stdout:3437: write oid 40 current snap is 449 2014-08-08T10:42:49.890 INFO:tasks.rados.rados.0.vpm167.stdout:3437: seq_num 1292 ranges {721816=636587,2144801=402051,3107685=14131} 2014-08-08T10:42:49.917 INFO:tasks.rados.rados.0.vpm167.stdout:3437: writing vpm1675861-40 from 721816 to 1358403 tid 1 2014-08-08T10:42:49.934 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 5, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:49.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:49.934 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 4, 3, 5, 2] 2014-08-08T10:42:49.935 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-08-08T10:42:49.950 INFO:tasks.rados.rados.0.vpm167.stdout:3437: writing vpm1675861-40 from 2144801 to 2546852 tid 2 2014-08-08T10:42:49.957 INFO:tasks.rados.rados.0.vpm167.stdout:3437: writing vpm1675861-40 from 3107685 to 3121816 tid 3 2014-08-08T10:42:49.958 INFO:tasks.rados.rados.0.vpm167.stdout:3438: read oid 22 snap -1 2014-08-08T10:42:49.959 INFO:tasks.rados.rados.0.vpm167.stdout:3439: snap_create 2014-08-08T10:42:50.117 INFO:tasks.rados.rados.0.vpm167.stdout:3437: finishing write tid 1 to vpm1675861-40 2014-08-08T10:42:50.121 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 50 v 792 (ObjNum 871 snap 293 seq_num 871) dirty exists 2014-08-08T10:42:50.125 INFO:tasks.rados.rados.0.vpm167.stdout:3437: finishing write tid 2 to vpm1675861-40 2014-08-08T10:42:50.129 INFO:tasks.rados.rados.0.vpm167.stdout:3437: finishing write tid 3 to vpm1675861-40 2014-08-08T10:42:50.131 INFO:tasks.rados.rados.0.vpm167.stdout:3437: finishing write tid 5 to vpm1675861-40 2014-08-08T10:42:50.132 INFO:tasks.rados.rados.0.vpm167.stdout:3437: finishing write tid 6 to vpm1675861-40 2014-08-08T10:42:50.132 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 40 v 903 (ObjNum 1292 snap 449 seq_num 1292) dirty exists 2014-08-08T10:42:51.006 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.5. 2014-08-08T10:42:51.085 INFO:tasks.rados.rados.0.vpm167.stdout:3436: done (3 left) 2014-08-08T10:42:51.086 INFO:tasks.rados.rados.0.vpm167.stdout:3437: done (2 left) 2014-08-08T10:42:51.086 INFO:tasks.rados.rados.0.vpm167.stdout:3438: done (1 left) 2014-08-08T10:42:51.086 INFO:tasks.rados.rados.0.vpm167.stdout:3439: done (0 left) 2014-08-08T10:42:51.086 INFO:tasks.rados.rados.0.vpm167.stdout:3440: write oid 31 current snap is 450 2014-08-08T10:42:51.086 INFO:tasks.rados.rados.0.vpm167.stdout:3440: seq_num 1293 ranges {514198=400000} 2014-08-08T10:42:51.103 INFO:tasks.rados.rados.0.vpm167.stdout:3440: writing vpm1675861-31 from 514198 to 914198 tid 1 2014-08-08T10:42:51.106 INFO:tasks.rados.rados.0.vpm167.stdout:3441: read oid 6 snap -1 2014-08-08T10:42:51.107 INFO:tasks.rados.rados.0.vpm167.stdout:3442: snap_create 2014-08-08T10:42:51.370 INFO:tasks.rados.rados.0.vpm167.stdout:3440: finishing write tid 1 to vpm1675861-31 2014-08-08T10:42:51.382 INFO:tasks.rados.rados.0.vpm167.stdout:3440: finishing write tid 3 to vpm1675861-31 2014-08-08T10:42:51.521 INFO:tasks.rados.rados.0.vpm167.stdout:3440: finishing write tid 4 to vpm1675861-31 2014-08-08T10:42:51.521 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1370 (ObjNum 1293 snap 450 seq_num 1293) dirty exists 2014-08-08T10:42:52.466 INFO:tasks.rados.rados.0.vpm167.stdout:3440: done (2 left) 2014-08-08T10:42:52.467 INFO:tasks.rados.rados.0.vpm167.stdout:3441: done (1 left) 2014-08-08T10:42:52.467 INFO:tasks.rados.rados.0.vpm167.stdout:3442: done (0 left) 2014-08-08T10:42:52.467 INFO:tasks.rados.rados.0.vpm167.stdout:3443: rollback oid 17 current snap is 451 2014-08-08T10:42:52.467 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 17 to 344 2014-08-08T10:42:52.468 INFO:tasks.rados.rados.0.vpm167.stdout:3444: snap_create 2014-08-08T10:42:53.726 INFO:tasks.rados.rados.0.vpm167.stdout:3444: done (1 left) 2014-08-08T10:42:53.726 INFO:tasks.rados.rados.0.vpm167.stdout:3445: snap_create 2014-08-08T10:42:54.819 INFO:tasks.rados.rados.0.vpm167.stdout:3445: done (1 left) 2014-08-08T10:42:54.819 INFO:tasks.rados.rados.0.vpm167.stdout:3446: write oid 26 current snap is 453 2014-08-08T10:42:54.820 INFO:tasks.rados.rados.0.vpm167.stdout:3446: seq_num 1294 ranges {726804=400000} 2014-08-08T10:42:54.840 INFO:tasks.rados.rados.0.vpm167.stdout:3446: writing vpm1675861-26 from 726804 to 1126804 tid 1 2014-08-08T10:42:54.843 INFO:tasks.rados.rados.0.vpm167.stdout:3447: delete oid 38 current snap is 453 2014-08-08T10:42:55.409 INFO:tasks.rados.rados.0.vpm167.stdout:3446: finishing write tid 1 to vpm1675861-26 2014-08-08T10:42:55.410 INFO:tasks.rados.rados.0.vpm167.stdout:3446: finishing write tid 3 to vpm1675861-26 2014-08-08T10:42:55.742 INFO:tasks.rados.rados.0.vpm167.stdout:3446: finishing write tid 4 to vpm1675861-26 2014-08-08T10:42:55.742 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 960 (ObjNum 1294 snap 453 seq_num 1294) dirty exists 2014-08-08T10:42:56.062 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2] out_osds: [0, 5] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:42:56.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:42:56.063 INFO:tasks.thrashosds.thrasher:Adding osd 0 2014-08-08T10:42:56.063 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 0' 2014-08-08T10:42:56.283 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2259 (ObjNum 57035360 snap 0 seq_num 32) dirty dne 2014-08-08T10:42:56.798 INFO:teuthology.orchestra.run.vpm166.stderr:marked in osd.0. 2014-08-08T10:42:56.844 INFO:tasks.thrashosds.thrasher:Added osd 0 2014-08-08T10:42:58.693 INFO:tasks.rados.rados.0.vpm167.stdout:3443: done (2 left) 2014-08-08T10:42:58.693 INFO:tasks.rados.rados.0.vpm167.stdout:3446: done (1 left) 2014-08-08T10:42:58.693 INFO:tasks.rados.rados.0.vpm167.stdout:3447: done (0 left) 2014-08-08T10:42:58.694 INFO:tasks.rados.rados.0.vpm167.stdout:3448: read oid 49 snap 450 2014-08-08T10:42:58.694 INFO:tasks.rados.rados.0.vpm167.stdout:3449: write oid 27 current snap is 453 2014-08-08T10:42:58.694 INFO:tasks.rados.rados.0.vpm167.stdout:3449: seq_num 1296 ranges {615439=711324,1978737=434256,2615438=1} 2014-08-08T10:42:58.726 INFO:tasks.rados.rados.0.vpm167.stdout:3449: writing vpm1675861-27 from 615439 to 1326763 tid 1 2014-08-08T10:42:58.759 INFO:tasks.rados.rados.0.vpm167.stdout:3449: writing vpm1675861-27 from 1978737 to 2412993 tid 2 2014-08-08T10:42:58.762 INFO:tasks.rados.rados.0.vpm167.stdout:3449: writing vpm1675861-27 from 2615438 to 2615439 tid 3 2014-08-08T10:42:58.763 INFO:tasks.rados.rados.0.vpm167.stdout:3448: expect (ObjNum 1273 snap 440 seq_num 1273) 2014-08-08T10:43:00.497 INFO:tasks.rados.rados.0.vpm167.stdout:3448: done (1 left) 2014-08-08T10:43:00.498 INFO:tasks.rados.rados.0.vpm167.stdout:3450: delete oid 13 current snap is 453 2014-08-08T10:43:00.499 INFO:tasks.rados.rados.0.vpm167.stdout:3449: finishing write tid 1 to vpm1675861-27 2014-08-08T10:43:00.499 INFO:tasks.rados.rados.0.vpm167.stdout:3449: finishing write tid 2 to vpm1675861-27 2014-08-08T10:43:00.499 INFO:tasks.rados.rados.0.vpm167.stdout:3449: finishing write tid 3 to vpm1675861-27 2014-08-08T10:43:00.500 INFO:tasks.rados.rados.0.vpm167.stdout:3449: finishing write tid 5 to vpm1675861-27 2014-08-08T10:43:00.500 INFO:tasks.rados.rados.0.vpm167.stdout:3449: finishing write tid 6 to vpm1675861-27 2014-08-08T10:43:00.500 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 27 v 2264 (ObjNum 1296 snap 453 seq_num 1296) dirty exists 2014-08-08T10:43:01.844 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:01.844 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:01.844 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-08T10:43:01.845 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:43:01.845 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:43:01.845 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-08T10:43:07.329 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:07.330 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T10:43:08.146 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T10:43:08.205 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-08T10:43:10.326 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T10:43:10.365 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T10:43:11.402 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T10:43:11.440 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T10:43:12.566 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T10:43:12.627 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-08T10:43:13.754 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.0 to 1 (8655362) 2014-08-08T10:43:13.818 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T10:43:13.819 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:14.731 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:14.799 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:15.036 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:15.106 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:16.456 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:42:19.622502', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88449900, u'health': u'HEALTH_OK', u'kb_used': 10503668, u'store_stats': {u'bytes_total': 32456249, u'bytes_log': 5602837, u'last_updated': u'0.000000', u'bytes_misc': 26853412, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:26.393339', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87316340, u'health': u'HEALTH_OK', u'kb_used': 11637228, u'store_stats': {u'bytes_total': 36096972, u'bytes_log': 9257588, u'last_updated': u'0.000000', u'bytes_misc': 26839384, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:23.352345', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88446140, u'health': u'HEALTH_OK', u'kb_used': 10507428, u'store_stats': {u'bytes_total': 35693429, u'bytes_log': 8833927, u'last_updated': u'0.000000', u'bytes_misc': 26859502, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 107/754 objects degraded (14.191%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 54 > pgp_num 44'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 4901432, u'degraded_objects': 107, u'num_pgs': 348, u'recovering_keys_per_sec': 0, u'data_bytes': 318700187, u'degraded_total': 754, u'bytes_used': 1621590016, u'recovering_objects_per_sec': 3, u'version': 2445, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 342, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.191', u'bytes_avail': 1071595945984}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1320, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:43:16.457 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:16.849 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:43:16.849 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:17.367 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:19.639 INFO:tasks.rados.rados.0.vpm167.stdout:3449: done (1 left) 2014-08-08T10:43:19.639 INFO:tasks.rados.rados.0.vpm167.stdout:3450: done (0 left) 2014-08-08T10:43:19.639 INFO:tasks.rados.rados.0.vpm167.stdout:3451: snap_create 2014-08-08T10:43:20.410 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:20.460 INFO:tasks.rados.rados.0.vpm167.stdout:3451: done (0 left) 2014-08-08T10:43:20.460 INFO:tasks.rados.rados.0.vpm167.stdout:3452: read oid 38 snap -1 2014-08-08T10:43:20.460 INFO:tasks.rados.rados.0.vpm167.stdout:3453: read oid 40 snap 384 2014-08-08T10:43:20.461 INFO:tasks.rados.rados.0.vpm167.stdout:3454: snap_remove snap 388 2014-08-08T10:43:20.593 INFO:tasks.rados.rados.0.vpm167.stdout:3453: expect (ObjNum 990 snap 335 seq_num 990) 2014-08-08T10:43:20.944 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:21.016 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:21.411 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:43:19.623148', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88369220, u'health': u'HEALTH_OK', u'kb_used': 10584348, u'store_stats': {u'bytes_total': 31044342, u'bytes_log': 2423285, u'last_updated': u'0.000000', u'bytes_misc': 28621057, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:26.393339', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87316340, u'health': u'HEALTH_OK', u'kb_used': 11637228, u'store_stats': {u'bytes_total': 36096972, u'bytes_log': 9257588, u'last_updated': u'0.000000', u'bytes_misc': 26839384, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:42:23.352345', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88446140, u'health': u'HEALTH_OK', u'kb_used': 10507428, u'store_stats': {u'bytes_total': 35693429, u'bytes_log': 8833927, u'last_updated': u'0.000000', u'bytes_misc': 26859502, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 97/754 objects degraded (12.865%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 54 > pgp_num 44'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 1047898, u'degraded_objects': 97, u'num_pgs': 348, u'recovering_keys_per_sec': 63, u'data_bytes': 318700187, u'degraded_total': 754, u'bytes_used': 1703424000, u'recovering_objects_per_sec': 3, u'version': 2449, u'pgs_by_state': [{u'count': 343, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.865', u'bytes_avail': 1071514112000}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1321, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:43:21.412 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:22.788 INFO:tasks.rados.rados.0.vpm167.stdout:3452: done (2 left) 2014-08-08T10:43:22.788 INFO:tasks.rados.rados.0.vpm167.stdout:3453: done (1 left) 2014-08-08T10:43:22.789 INFO:tasks.rados.rados.0.vpm167.stdout:3454: done (0 left) 2014-08-08T10:43:22.789 INFO:tasks.rados.rados.0.vpm167.stdout:3455: snap_remove snap 447 2014-08-08T10:43:23.151 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:43:23.151 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:24.069 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:24.101 INFO:tasks.rados.rados.0.vpm167.stdout:3455: done (0 left) 2014-08-08T10:43:24.101 INFO:tasks.rados.rados.0.vpm167.stdout:3456: snap_create 2014-08-08T10:43:25.135 INFO:tasks.rados.rados.0.vpm167.stdout:3456: done (0 left) 2014-08-08T10:43:25.135 INFO:tasks.rados.rados.0.vpm167.stdout:3457: delete oid 11 current snap is 455 2014-08-08T10:43:26.544 INFO:tasks.rados.rados.0.vpm167.stdout:3457: done (0 left) 2014-08-08T10:43:26.544 INFO:tasks.rados.rados.0.vpm167.stdout:3458: read oid 23 snap -1 2014-08-08T10:43:26.544 INFO:tasks.rados.rados.0.vpm167.stdout:3459: snap_create 2014-08-08T10:43:26.552 INFO:tasks.rados.rados.0.vpm167.stdout:3458: expect (ObjNum 1235 snap 432 seq_num 1235) 2014-08-08T10:43:26.954 INFO:tasks.rados.rados.0.vpm167.stdout:3458: done (1 left) 2014-08-08T10:43:26.955 INFO:tasks.rados.rados.0.vpm167.stdout:3459: done (0 left) 2014-08-08T10:43:26.955 INFO:tasks.rados.rados.0.vpm167.stdout:3460: read oid 8 snap 440 2014-08-08T10:43:26.955 INFO:tasks.rados.rados.0.vpm167.stdout:3461: write oid 4 current snap is 456 2014-08-08T10:43:26.955 INFO:tasks.rados.rados.0.vpm167.stdout:3461: seq_num 1299 ranges {315655=1} 2014-08-08T10:43:26.958 INFO:tasks.rados.rados.0.vpm167.stdout:3461: writing vpm1675861-4 from 315655 to 315656 tid 1 2014-08-08T10:43:26.960 INFO:tasks.rados.rados.0.vpm167.stdout:3462: read oid 46 snap -1 2014-08-08T10:43:26.960 INFO:tasks.rados.rados.0.vpm167.stdout:3463: read oid 43 snap 415 2014-08-08T10:43:26.961 INFO:tasks.rados.rados.0.vpm167.stdout:3464: write oid 26 current snap is 456 2014-08-08T10:43:26.961 INFO:tasks.rados.rados.0.vpm167.stdout:3464: seq_num 1300 ranges {11909=1} 2014-08-08T10:43:26.962 INFO:tasks.rados.rados.0.vpm167.stdout:3464: writing vpm1675861-26 from 11909 to 11910 tid 1 2014-08-08T10:43:26.963 INFO:tasks.rados.rados.0.vpm167.stdout:3465: read oid 2 snap -1 2014-08-08T10:43:26.963 INFO:tasks.rados.rados.0.vpm167.stdout:3466: write oid 12 current snap is 456 2014-08-08T10:43:26.964 INFO:tasks.rados.rados.0.vpm167.stdout:3466: seq_num 1301 ranges {507140=509290,1676183=571728,2711994=549731,3707139=1} 2014-08-08T10:43:27.014 INFO:tasks.rados.rados.0.vpm167.stdout:3466: writing vpm1675861-12 from 507140 to 1016430 tid 1 2014-08-08T10:43:27.078 INFO:tasks.rados.rados.0.vpm167.stdout:3466: writing vpm1675861-12 from 1676183 to 2247911 tid 2 2014-08-08T10:43:27.115 INFO:tasks.rados.rados.0.vpm167.stdout:3466: writing vpm1675861-12 from 2711994 to 3261725 tid 3 2014-08-08T10:43:27.128 INFO:tasks.rados.rados.0.vpm167.stdout:3466: writing vpm1675861-12 from 3707139 to 3707140 tid 4 2014-08-08T10:43:27.128 INFO:tasks.rados.rados.0.vpm167.stdout:3465: expect (ObjNum 1101 snap 385 seq_num 1101) 2014-08-08T10:43:27.179 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:27.549 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:27.617 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:27.943 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:43:19.623148', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88369220, u'health': u'HEALTH_OK', u'kb_used': 10584348, u'store_stats': {u'bytes_total': 31044342, u'bytes_log': 2423285, u'last_updated': u'0.000000', u'bytes_misc': 28621057, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:43:26.393708', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87245288, u'health': u'HEALTH_OK', u'kb_used': 11708280, u'store_stats': {u'bytes_total': 26774552, u'bytes_log': 678433, u'last_updated': u'0.000000', u'bytes_misc': 26096119, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:43:23.352745', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88351144, u'health': u'HEALTH_OK', u'kb_used': 10602424, u'store_stats': {u'bytes_total': 39001987, u'bytes_log': 161915, u'last_updated': u'0.000000', u'bytes_misc': 38840072, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 46/754 objects degraded (6.101%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 54 > pgp_num 44'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 20762778, u'degraded_objects': 46, u'num_pgs': 348, u'recovering_keys_per_sec': 0, u'data_bytes': 318700187, u'degraded_total': 754, u'bytes_used': 1718120448, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 2454, u'pgs_by_state': [{u'count': 344, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.101', u'bytes_avail': 1071499415552}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1325, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:43:27.944 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:28.197 INFO:tasks.rados.rados.0.vpm167.stdout:3465: done (6 left) 2014-08-08T10:43:28.198 INFO:tasks.rados.rados.0.vpm167.stdout:3467: read oid 48 snap -1 2014-08-08T10:43:28.199 INFO:tasks.rados.rados.0.vpm167.stdout:3468: delete oid 20 current snap is 456 2014-08-08T10:43:28.199 INFO:tasks.rados.rados.0.vpm167.stdout:3463: expect (ObjNum 1140 snap 403 seq_num 1140) 2014-08-08T10:43:28.356 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:43:28.356 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:28.572 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:28.819 INFO:tasks.rados.rados.0.vpm167.stdout:3463: done (7 left) 2014-08-08T10:43:28.819 INFO:tasks.rados.rados.0.vpm167.stdout:3468: done (6 left) 2014-08-08T10:43:28.819 INFO:tasks.rados.rados.0.vpm167.stdout:3469: delete oid 47 current snap is 456 2014-08-08T10:43:28.820 INFO:tasks.rados.rados.0.vpm167.stdout:3460: expect (ObjNum 1063 snap 368 seq_num 1063) 2014-08-08T10:43:30.006 INFO:tasks.rados.rados.0.vpm167.stdout:3460: done (6 left) 2014-08-08T10:43:30.006 INFO:tasks.rados.rados.0.vpm167.stdout:3469: done (5 left) 2014-08-08T10:43:30.007 INFO:tasks.rados.rados.0.vpm167.stdout:3470: delete oid 11 current snap is 456 2014-08-08T10:43:30.007 INFO:tasks.rados.rados.0.vpm167.stdout:3461: finishing write tid 1 to vpm1675861-4 2014-08-08T10:43:30.007 INFO:tasks.rados.rados.0.vpm167.stdout:3461: finishing write tid 3 to vpm1675861-4 2014-08-08T10:43:30.007 INFO:tasks.rados.rados.0.vpm167.stdout:3462: expect (ObjNum 1060 snap 364 seq_num 1060) 2014-08-08T10:43:30.648 INFO:tasks.rados.rados.0.vpm167.stdout:3462: done (5 left) 2014-08-08T10:43:30.648 INFO:tasks.rados.rados.0.vpm167.stdout:3470: done (4 left) 2014-08-08T10:43:30.649 INFO:tasks.rados.rados.0.vpm167.stdout:3471: snap_create 2014-08-08T10:43:30.649 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 1 to vpm1675861-12 2014-08-08T10:43:30.649 INFO:tasks.rados.rados.0.vpm167.stdout:3464: finishing write tid 1 to vpm1675861-26 2014-08-08T10:43:30.649 INFO:tasks.rados.rados.0.vpm167.stdout:3461: finishing write tid 4 to vpm1675861-4 2014-08-08T10:43:30.650 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 4 v 1266 (ObjNum 1299 snap 456 seq_num 1299) dirty exists 2014-08-08T10:43:30.650 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 2 to vpm1675861-12 2014-08-08T10:43:30.650 INFO:tasks.rados.rados.0.vpm167.stdout:3464: finishing write tid 3 to vpm1675861-26 2014-08-08T10:43:30.650 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 3 to vpm1675861-12 2014-08-08T10:43:30.650 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 4 to vpm1675861-12 2014-08-08T10:43:30.651 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 6 to vpm1675861-12 2014-08-08T10:43:30.651 INFO:tasks.rados.rados.0.vpm167.stdout:3466: finishing write tid 7 to vpm1675861-12 2014-08-08T10:43:30.651 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 12 v 2282 (ObjNum 1301 snap 456 seq_num 1301) dirty exists 2014-08-08T10:43:30.651 INFO:tasks.rados.rados.0.vpm167.stdout:3464: finishing write tid 4 to vpm1675861-26 2014-08-08T10:43:30.651 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 26 v 967 (ObjNum 1300 snap 456 seq_num 1300) dirty exists 2014-08-08T10:43:30.652 INFO:tasks.rados.rados.0.vpm167.stdout:3467: expect (ObjNum 1269 snap 440 seq_num 1269) 2014-08-08T10:43:31.212 INFO:tasks.rados.rados.0.vpm167.stdout:3461: done (4 left) 2014-08-08T10:43:31.213 INFO:tasks.rados.rados.0.vpm167.stdout:3464: done (3 left) 2014-08-08T10:43:31.213 INFO:tasks.rados.rados.0.vpm167.stdout:3466: done (2 left) 2014-08-08T10:43:31.213 INFO:tasks.rados.rados.0.vpm167.stdout:3467: done (1 left) 2014-08-08T10:43:31.214 INFO:tasks.rados.rados.0.vpm167.stdout:3471: done (0 left) 2014-08-08T10:43:31.214 INFO:tasks.rados.rados.0.vpm167.stdout:3472: write oid 17 current snap is 457 2014-08-08T10:43:31.214 INFO:tasks.rados.rados.0.vpm167.stdout:3472: seq_num 1305 ranges {553211=432603,1511143=603933,2153210=1} 2014-08-08T10:43:31.239 INFO:tasks.rados.rados.0.vpm167.stdout:3472: writing vpm1675861-17 from 553211 to 985814 tid 1 2014-08-08T10:43:31.281 INFO:tasks.rados.rados.0.vpm167.stdout:3472: writing vpm1675861-17 from 1511143 to 2115076 tid 2 2014-08-08T10:43:31.283 INFO:tasks.rados.rados.0.vpm167.stdout:3472: writing vpm1675861-17 from 2153210 to 2153211 tid 3 2014-08-08T10:43:31.284 INFO:tasks.rados.rados.0.vpm167.stdout:3473: snap_remove snap 443 2014-08-08T10:43:31.287 INFO:tasks.rados.rados.0.vpm167.stdout:3472: finishing write tid 1 to vpm1675861-17 2014-08-08T10:43:31.320 INFO:tasks.rados.rados.0.vpm167.stdout:3472: finishing write tid 2 to vpm1675861-17 2014-08-08T10:43:31.321 INFO:tasks.rados.rados.0.vpm167.stdout:3472: finishing write tid 3 to vpm1675861-17 2014-08-08T10:43:31.321 INFO:tasks.rados.rados.0.vpm167.stdout:3472: finishing write tid 5 to vpm1675861-17 2014-08-08T10:43:31.323 INFO:tasks.rados.rados.0.vpm167.stdout:3472: finishing write tid 6 to vpm1675861-17 2014-08-08T10:43:31.323 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2287 (ObjNum 1305 snap 457 seq_num 1305) dirty exists 2014-08-08T10:43:31.608 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:43:31.824 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:43:31.889 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:43:32.427 INFO:tasks.rados.rados.0.vpm167.stdout:3472: done (1 left) 2014-08-08T10:43:32.427 INFO:tasks.rados.rados.0.vpm167.stdout:3473: done (0 left) 2014-08-08T10:43:32.427 INFO:tasks.rados.rados.0.vpm167.stdout:3474: read oid 32 snap -1 2014-08-08T10:43:32.428 INFO:tasks.rados.rados.0.vpm167.stdout:3475: read oid 29 snap 445 2014-08-08T10:43:32.428 INFO:tasks.rados.rados.0.vpm167.stdout:3476: snap_remove snap 437 2014-08-08T10:43:32.457 INFO:tasks.rados.rados.0.vpm167.stdout:3474: expect (ObjNum 1212 snap 423 seq_num 1212) 2014-08-08T10:43:32.481 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:43:19.623148', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88369220, u'health': u'HEALTH_OK', u'kb_used': 10584348, u'store_stats': {u'bytes_total': 31044342, u'bytes_log': 2423285, u'last_updated': u'0.000000', u'bytes_misc': 28621057, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:43:26.393708', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87245288, u'health': u'HEALTH_OK', u'kb_used': 11708280, u'store_stats': {u'bytes_total': 26774552, u'bytes_log': 678433, u'last_updated': u'0.000000', u'bytes_misc': 26096119, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:43:23.352745', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88351144, u'health': u'HEALTH_OK', u'kb_used': 10602424, u'store_stats': {u'bytes_total': 39001987, u'bytes_log': 161915, u'last_updated': u'0.000000', u'bytes_misc': 38840072, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 54 > pgp_num 44'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'num_pgs': 348, u'data_bytes': 321015262, u'bytes_used': 1700601856, u'op_per_sec': 0, u'version': 2460, u'pgs_by_state': [{u'count': 348, u'state_name': u'active+clean'}], u'write_bytes_sec': 0, u'bytes_avail': 1071516934144}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1327, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:43:32.482 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T10:43:32.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:32.482 INFO:tasks.thrashosds.thrasher:Adding osd 5 2014-08-08T10:43:32.483 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-08-08T10:43:33.442 INFO:teuthology.orchestra.run.vpm166.stderr:marked in osd.5. 2014-08-08T10:43:33.492 INFO:tasks.thrashosds.thrasher:Added osd 5 2014-08-08T10:43:33.722 INFO:tasks.rados.rados.0.vpm167.stdout:3474: done (2 left) 2014-08-08T10:43:33.723 INFO:tasks.rados.rados.0.vpm167.stdout:3475: done (1 left) 2014-08-08T10:43:33.723 INFO:tasks.rados.rados.0.vpm167.stdout:3476: done (0 left) 2014-08-08T10:43:33.723 INFO:tasks.rados.rados.0.vpm167.stdout:3477: rollback oid 15 current snap is 457 2014-08-08T10:43:33.723 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 15 to 408 2014-08-08T10:43:33.724 INFO:tasks.rados.rados.0.vpm167.stdout:3478: rollback oid 3 current snap is 457 2014-08-08T10:43:33.724 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 445 2014-08-08T10:43:33.724 INFO:tasks.rados.rados.0.vpm167.stdout:3479: write oid 44 current snap is 457 2014-08-08T10:43:33.724 INFO:tasks.rados.rados.0.vpm167.stdout:3479: seq_num 1306 ranges {495992=405431,1472826=709200,2748519=147473} 2014-08-08T10:43:33.741 INFO:tasks.rados.rados.0.vpm167.stdout:3479: writing vpm1675861-44 from 495992 to 901423 tid 1 2014-08-08T10:43:33.774 INFO:tasks.rados.rados.0.vpm167.stdout:3479: writing vpm1675861-44 from 1472826 to 2182026 tid 2 2014-08-08T10:43:33.785 INFO:tasks.rados.rados.0.vpm167.stdout:3479: writing vpm1675861-44 from 2748519 to 2895992 tid 3 2014-08-08T10:43:33.786 INFO:tasks.rados.rados.0.vpm167.stdout:3480: delete oid 40 current snap is 457 2014-08-08T10:43:33.961 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 884 (ObjNum 1283 snap 445 seq_num 1283) dirty exists 2014-08-08T10:43:34.362 INFO:tasks.rados.rados.0.vpm167.stdout:3478: done (3 left) 2014-08-08T10:43:34.362 INFO:tasks.rados.rados.0.vpm167.stdout:3480: done (2 left) 2014-08-08T10:43:34.363 INFO:tasks.rados.rados.0.vpm167.stdout:3481: write oid 45 current snap is 457 2014-08-08T10:43:34.363 INFO:tasks.rados.rados.0.vpm167.stdout:3481: seq_num 1308 ranges {653428=785136,2066019=449590,2999547=781521,3853427=1} 2014-08-08T10:43:34.407 INFO:tasks.rados.rados.0.vpm167.stdout:3481: writing vpm1675861-45 from 653428 to 1438564 tid 1 2014-08-08T10:43:34.439 INFO:tasks.rados.rados.0.vpm167.stdout:3481: writing vpm1675861-45 from 2066019 to 2515609 tid 2 2014-08-08T10:43:34.471 INFO:tasks.rados.rados.0.vpm167.stdout:3481: writing vpm1675861-45 from 2999547 to 3781068 tid 3 2014-08-08T10:43:34.473 INFO:tasks.rados.rados.0.vpm167.stdout:3481: writing vpm1675861-45 from 3853427 to 3853428 tid 4 2014-08-08T10:43:34.473 INFO:tasks.rados.rados.0.vpm167.stdout:3479: finishing write tid 1 to vpm1675861-44 2014-08-08T10:43:34.473 INFO:tasks.rados.rados.0.vpm167.stdout:3482: snap_remove snap 444 2014-08-08T10:43:34.638 INFO:tasks.rados.rados.0.vpm167.stdout:3479: finishing write tid 2 to vpm1675861-44 2014-08-08T10:43:34.742 INFO:tasks.rados.rados.0.vpm167.stdout:3479: finishing write tid 3 to vpm1675861-44 2014-08-08T10:43:34.745 INFO:tasks.rados.rados.0.vpm167.stdout:3479: finishing write tid 5 to vpm1675861-44 2014-08-08T10:43:34.746 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 1 to vpm1675861-45 2014-08-08T10:43:34.746 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 2 to vpm1675861-45 2014-08-08T10:43:34.747 INFO:tasks.rados.rados.0.vpm167.stdout:3479: finishing write tid 6 to vpm1675861-44 2014-08-08T10:43:34.747 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 44 v 1532 (ObjNum 1306 snap 457 seq_num 1306) dirty exists 2014-08-08T10:43:34.759 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 3 to vpm1675861-45 2014-08-08T10:43:34.760 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 4 to vpm1675861-45 2014-08-08T10:43:34.760 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 6 to vpm1675861-45 2014-08-08T10:43:34.864 INFO:tasks.rados.rados.0.vpm167.stdout:3481: finishing write tid 7 to vpm1675861-45 2014-08-08T10:43:34.865 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 45 v 1540 (ObjNum 1308 snap 457 seq_num 1308) dirty exists 2014-08-08T10:43:34.909 INFO:tasks.rados.rados.0.vpm167.stdout:3479: done (3 left) 2014-08-08T10:43:34.909 INFO:tasks.rados.rados.0.vpm167.stdout:3481: done (2 left) 2014-08-08T10:43:34.909 INFO:tasks.rados.rados.0.vpm167.stdout:3482: done (1 left) 2014-08-08T10:43:34.909 INFO:tasks.rados.rados.0.vpm167.stdout:3483: read oid 47 snap 422 2014-08-08T10:43:34.910 INFO:tasks.rados.rados.0.vpm167.stdout:3484: delete oid 43 current snap is 457 2014-08-08T10:43:35.109 INFO:tasks.rados.rados.0.vpm167.stdout:3483: expect (ObjNum 1202 snap 421 seq_num 1202) 2014-08-08T10:43:38.491 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:38.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:38.491 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2014-08-08T10:43:38.492 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 2 0' 2014-08-08T10:43:38.912 INFO:tasks.rados.rados.0.vpm167.stdout:3483: done (2 left) 2014-08-08T10:43:38.912 INFO:tasks.rados.rados.0.vpm167.stdout:3484: done (1 left) 2014-08-08T10:43:38.918 INFO:tasks.rados.rados.0.vpm167.stdout:3485: write oid 3 current snap is 457 2014-08-08T10:43:38.918 INFO:tasks.rados.rados.0.vpm167.stdout:3485: seq_num 1310 ranges {542063=481193,1342062=1} 2014-08-08T10:43:38.943 INFO:tasks.rados.rados.0.vpm167.stdout:3485: writing vpm1675861-3 from 542063 to 1023256 tid 1 2014-08-08T10:43:38.962 INFO:tasks.rados.rados.0.vpm167.stdout:3485: writing vpm1675861-3 from 1342062 to 1342063 tid 2 2014-08-08T10:43:38.963 INFO:tasks.rados.rados.0.vpm167.stdout:3486: delete oid 33 current snap is 457 2014-08-08T10:43:38.990 INFO:tasks.rados.rados.0.vpm167.stdout:3485: finishing write tid 1 to vpm1675861-3 2014-08-08T10:43:39.113 INFO:tasks.rados.rados.0.vpm167.stdout:3485: finishing write tid 2 to vpm1675861-3 2014-08-08T10:43:39.122 INFO:tasks.rados.rados.0.vpm167.stdout:3485: finishing write tid 4 to vpm1675861-3 2014-08-08T10:43:39.138 INFO:tasks.rados.rados.0.vpm167.stdout:3485: finishing write tid 5 to vpm1675861-3 2014-08-08T10:43:39.139 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 889 (ObjNum 1310 snap 457 seq_num 1310) dirty exists 2014-08-08T10:43:39.256 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 15 v 1893 (ObjNum 48213448 snap 0 seq_num 48194384) dirty dne 2014-08-08T10:43:39.782 INFO:teuthology.orchestra.run.vpm166.stderr:set osd.2 primary-affinity to 0 (802) 2014-08-08T10:43:40.034 INFO:tasks.rados.rados.0.vpm167.stdout:3477: done (2 left) 2014-08-08T10:43:40.035 INFO:tasks.rados.rados.0.vpm167.stdout:3485: done (1 left) 2014-08-08T10:43:40.035 INFO:tasks.rados.rados.0.vpm167.stdout:3486: done (0 left) 2014-08-08T10:43:40.035 INFO:tasks.rados.rados.0.vpm167.stdout:3487: read oid 35 snap 434 2014-08-08T10:43:40.035 INFO:tasks.rados.rados.0.vpm167.stdout:3488: snap_remove snap 419 2014-08-08T10:43:40.916 INFO:tasks.rados.rados.0.vpm167.stdout:3487: done (1 left) 2014-08-08T10:43:40.916 INFO:tasks.rados.rados.0.vpm167.stdout:3488: done (0 left) 2014-08-08T10:43:40.916 INFO:tasks.rados.rados.0.vpm167.stdout:3489: snap_create 2014-08-08T10:43:41.939 INFO:tasks.rados.rados.0.vpm167.stdout:3489: done (0 left) 2014-08-08T10:43:41.939 INFO:tasks.rados.rados.0.vpm167.stdout:3490: snap_create 2014-08-08T10:43:43.143 INFO:tasks.rados.rados.0.vpm167.stdout:3490: done (0 left) 2014-08-08T10:43:43.144 INFO:tasks.rados.rados.0.vpm167.stdout:3491: snap_remove snap 451 2014-08-08T10:43:44.328 INFO:tasks.rados.rados.0.vpm167.stdout:3491: done (0 left) 2014-08-08T10:43:44.328 INFO:tasks.rados.rados.0.vpm167.stdout:3492: delete oid 4 current snap is 459 2014-08-08T10:43:44.438 INFO:tasks.rados.rados.0.vpm167.stdout:3492: done (0 left) 2014-08-08T10:43:44.438 INFO:tasks.rados.rados.0.vpm167.stdout:3493: write oid 40 current snap is 459 2014-08-08T10:43:44.439 INFO:tasks.rados.rados.0.vpm167.stdout:3493: seq_num 1313 ranges {643305=432702,1863688=702390,3043304=1} 2014-08-08T10:43:44.470 INFO:tasks.rados.rados.0.vpm167.stdout:3493: writing vpm1675861-40 from 643305 to 1076007 tid 1 2014-08-08T10:43:44.519 INFO:tasks.rados.rados.0.vpm167.stdout:3493: writing vpm1675861-40 from 1863688 to 2566078 tid 2 2014-08-08T10:43:44.526 INFO:tasks.rados.rados.0.vpm167.stdout:3493: writing vpm1675861-40 from 3043304 to 3043305 tid 3 2014-08-08T10:43:44.526 INFO:tasks.rados.rados.0.vpm167.stdout:3494: delete oid 18 current snap is 459 2014-08-08T10:43:44.734 INFO:tasks.rados.rados.0.vpm167.stdout:3493: finishing write tid 1 to vpm1675861-40 2014-08-08T10:43:44.735 INFO:tasks.rados.rados.0.vpm167.stdout:3493: finishing write tid 2 to vpm1675861-40 2014-08-08T10:43:44.741 INFO:tasks.rados.rados.0.vpm167.stdout:3493: finishing write tid 3 to vpm1675861-40 2014-08-08T10:43:44.743 INFO:tasks.rados.rados.0.vpm167.stdout:3493: finishing write tid 5 to vpm1675861-40 2014-08-08T10:43:44.743 INFO:tasks.rados.rados.0.vpm167.stdout:3494: done (1 left) 2014-08-08T10:43:44.744 INFO:tasks.rados.rados.0.vpm167.stdout:3495: read oid 46 snap 455 2014-08-08T10:43:44.744 INFO:tasks.rados.rados.0.vpm167.stdout:3496: snap_remove snap 396 2014-08-08T10:43:44.762 INFO:tasks.rados.rados.0.vpm167.stdout:3493: finishing write tid 6 to vpm1675861-40 2014-08-08T10:43:44.762 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 40 v 927 (ObjNum 1313 snap 459 seq_num 1313) dirty exists 2014-08-08T10:43:44.789 INFO:tasks.rados.rados.0.vpm167.stdout:3495: expect (ObjNum 1060 snap 364 seq_num 1060) 2014-08-08T10:43:44.812 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:44.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:44.813 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T10:43:44.813 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:43:44.814 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:43:44.814 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T10:43:45.803 INFO:tasks.rados.rados.0.vpm167.stdout:3493: done (2 left) 2014-08-08T10:43:45.803 INFO:tasks.rados.rados.0.vpm167.stdout:3495: done (1 left) 2014-08-08T10:43:45.804 INFO:tasks.rados.rados.0.vpm167.stdout:3496: done (0 left) 2014-08-08T10:43:45.804 INFO:tasks.rados.rados.0.vpm167.stdout:3497: read oid 40 snap 450 2014-08-08T10:43:45.805 INFO:tasks.rados.rados.0.vpm167.stdout:3498: write oid 28 current snap is 459 2014-08-08T10:43:45.805 INFO:tasks.rados.rados.0.vpm167.stdout:3498: seq_num 1315 ranges {682340=527735,1805232=726429,3152829=329511} 2014-08-08T10:43:45.852 INFO:tasks.rados.rados.0.vpm167.stdout:3498: writing vpm1675861-28 from 682340 to 1210075 tid 1 2014-08-08T10:43:45.899 INFO:tasks.rados.rados.0.vpm167.stdout:3498: writing vpm1675861-28 from 1805232 to 2531661 tid 2 2014-08-08T10:43:45.924 INFO:tasks.rados.rados.0.vpm167.stdout:3498: writing vpm1675861-28 from 3152829 to 3482340 tid 3 2014-08-08T10:43:45.926 INFO:tasks.rados.rados.0.vpm167.stdout:3497: expect (ObjNum 1292 snap 449 seq_num 1292) 2014-08-08T10:43:47.397 INFO:tasks.rados.rados.0.vpm167.stdout:3497: done (1 left) 2014-08-08T10:43:47.398 INFO:tasks.rados.rados.0.vpm167.stdout:3499: delete oid 42 current snap is 459 2014-08-08T10:43:47.399 INFO:tasks.rados.rados.0.vpm167.stdout:3498: finishing write tid 1 to vpm1675861-28 2014-08-08T10:43:47.399 INFO:tasks.rados.rados.0.vpm167.stdout:3498: finishing write tid 2 to vpm1675861-28 2014-08-08T10:43:47.399 INFO:tasks.rados.rados.0.vpm167.stdout:3498: finishing write tid 3 to vpm1675861-28 2014-08-08T10:43:47.399 INFO:tasks.rados.rados.0.vpm167.stdout:3498: finishing write tid 5 to vpm1675861-28 2014-08-08T10:43:47.399 INFO:tasks.rados.rados.0.vpm167.stdout:3498: finishing write tid 6 to vpm1675861-28 2014-08-08T10:43:47.400 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 811 (ObjNum 1315 snap 459 seq_num 1315) dirty exists 2014-08-08T10:43:47.400 INFO:tasks.rados.rados.0.vpm167.stdout:3498: done (1 left) 2014-08-08T10:43:47.401 INFO:tasks.rados.rados.0.vpm167.stdout:3499: done (0 left) 2014-08-08T10:43:47.402 INFO:tasks.rados.rados.0.vpm167.stdout:3500: rollback oid 3 current snap is 459 2014-08-08T10:43:47.402 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 445 2014-08-08T10:43:47.402 INFO:tasks.rados.rados.0.vpm167.stdout:3501: rollback oid 20 current snap is 459 2014-08-08T10:43:47.403 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 20 to 446 2014-08-08T10:43:47.403 INFO:tasks.rados.rados.0.vpm167.stdout:3502: snap_remove snap 435 2014-08-08T10:43:47.414 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 20 v 992 (ObjNum 1041 snap 358 seq_num 1041) dirty exists 2014-08-08T10:43:47.417 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 897 (ObjNum 1283 snap 445 seq_num 1283) dirty exists 2014-08-08T10:43:48.033 INFO:tasks.rados.rados.0.vpm167.stdout:3500: done (2 left) 2014-08-08T10:43:48.033 INFO:tasks.rados.rados.0.vpm167.stdout:3501: done (1 left) 2014-08-08T10:43:48.034 INFO:tasks.rados.rados.0.vpm167.stdout:3502: done (0 left) 2014-08-08T10:43:48.034 INFO:tasks.rados.rados.0.vpm167.stdout:3503: delete oid 34 current snap is 459 2014-08-08T10:43:48.041 INFO:tasks.rados.rados.0.vpm167.stdout:3503: done (0 left) 2014-08-08T10:43:48.041 INFO:tasks.rados.rados.0.vpm167.stdout:3504: write oid 30 current snap is 459 2014-08-08T10:43:48.042 INFO:tasks.rados.rados.0.vpm167.stdout:3504: seq_num 1318 ranges {135004=1} 2014-08-08T10:43:48.043 INFO:tasks.rados.rados.0.vpm167.stdout:3504: writing vpm1675861-30 from 135004 to 135005 tid 1 2014-08-08T10:43:48.045 INFO:tasks.rados.rados.0.vpm167.stdout:3505: delete oid 3 current snap is 459 2014-08-08T10:43:48.077 INFO:tasks.rados.rados.0.vpm167.stdout:3504: finishing write tid 1 to vpm1675861-30 2014-08-08T10:43:48.244 INFO:tasks.rados.rados.0.vpm167.stdout:3504: finishing write tid 3 to vpm1675861-30 2014-08-08T10:43:48.248 INFO:tasks.rados.rados.0.vpm167.stdout:3505: done (1 left) 2014-08-08T10:43:48.248 INFO:tasks.rados.rados.0.vpm167.stdout:3506: write oid 16 current snap is 459 2014-08-08T10:43:48.249 INFO:tasks.rados.rados.0.vpm167.stdout:3506: seq_num 1320 ranges {423641=458948,1623640=1} 2014-08-08T10:43:48.266 INFO:tasks.rados.rados.0.vpm167.stdout:3506: writing vpm1675861-16 from 423641 to 882589 tid 1 2014-08-08T10:43:48.279 INFO:tasks.rados.rados.0.vpm167.stdout:3506: writing vpm1675861-16 from 1623640 to 1623641 tid 2 2014-08-08T10:43:48.280 INFO:tasks.rados.rados.0.vpm167.stdout:3507: read oid 35 snap -1 2014-08-08T10:43:48.281 INFO:tasks.rados.rados.0.vpm167.stdout:3508: rollback oid 2 current snap is 459 2014-08-08T10:43:48.281 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 2 to 448 2014-08-08T10:43:48.282 INFO:tasks.rados.rados.0.vpm167.stdout:3509: read oid 7 snap -1 2014-08-08T10:43:48.282 INFO:tasks.rados.rados.0.vpm167.stdout:3510: snap_create 2014-08-08T10:43:48.310 INFO:tasks.rados.rados.0.vpm167.stdout:3507: expect (ObjNum 1289 snap 448 seq_num 1289) 2014-08-08T10:43:49.117 INFO:tasks.rados.rados.0.vpm167.stdout:3507: done (5 left) 2014-08-08T10:43:49.118 INFO:tasks.rados.rados.0.vpm167.stdout:3510: done (4 left) 2014-08-08T10:43:49.119 INFO:tasks.rados.rados.0.vpm167.stdout:3511: read oid 37 snap -1 2014-08-08T10:43:49.119 INFO:tasks.rados.rados.0.vpm167.stdout:3512: rollback oid 41 current snap is 460 2014-08-08T10:43:49.119 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 41 to 446 2014-08-08T10:43:49.120 INFO:tasks.rados.rados.0.vpm167.stdout:3513: write oid 17 current snap is 460 2014-08-08T10:43:49.120 INFO:tasks.rados.rados.0.vpm167.stdout:3513: seq_num 1321 ranges {635222=582177,1618857=216365} 2014-08-08T10:43:49.155 INFO:tasks.rados.rados.0.vpm167.stdout:3513: writing vpm1675861-17 from 635222 to 1217399 tid 1 2014-08-08T10:43:49.176 INFO:tasks.rados.rados.0.vpm167.stdout:3513: writing vpm1675861-17 from 1618857 to 1835222 tid 2 2014-08-08T10:43:49.177 INFO:tasks.rados.rados.0.vpm167.stdout:3509: expect (ObjNum 1279 snap 441 seq_num 1279) 2014-08-08T10:43:49.933 INFO:tasks.rados.rados.0.vpm167.stdout:3509: done (6 left) 2014-08-08T10:43:49.933 INFO:tasks.rados.rados.0.vpm167.stdout:3514: snap_create 2014-08-08T10:43:49.934 INFO:tasks.rados.rados.0.vpm167.stdout:3506: finishing write tid 1 to vpm1675861-16 2014-08-08T10:43:49.934 INFO:tasks.rados.rados.0.vpm167.stdout:3506: finishing write tid 2 to vpm1675861-16 2014-08-08T10:43:49.934 INFO:tasks.rados.rados.0.vpm167.stdout:3506: finishing write tid 4 to vpm1675861-16 2014-08-08T10:43:49.935 INFO:tasks.rados.rados.0.vpm167.stdout:3504: finishing write tid 4 to vpm1675861-30 2014-08-08T10:43:49.935 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 30 v 1306 (ObjNum 1318 snap 459 seq_num 1318) dirty exists 2014-08-08T10:43:49.935 INFO:tasks.rados.rados.0.vpm167.stdout:3506: finishing write tid 5 to vpm1675861-16 2014-08-08T10:43:49.935 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1564 (ObjNum 1320 snap 459 seq_num 1320) dirty exists 2014-08-08T10:43:49.935 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 2 v 1439 (ObjNum 1101 snap 385 seq_num 1101) dirty exists 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1865 (ObjNum 1274 snap 440 seq_num 1274) dirty exists 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:3513: finishing write tid 1 to vpm1675861-17 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:3513: finishing write tid 2 to vpm1675861-17 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:3513: finishing write tid 4 to vpm1675861-17 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:3513: finishing write tid 5 to vpm1675861-17 2014-08-08T10:43:49.936 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2317 (ObjNum 1321 snap 460 seq_num 1321) dirty exists 2014-08-08T10:43:50.151 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:50.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:50.152 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-08T10:43:50.152 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:43:50.152 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:43:50.152 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-08T10:43:51.420 INFO:tasks.rados.rados.0.vpm167.stdout:3504: done (6 left) 2014-08-08T10:43:51.422 INFO:tasks.rados.rados.0.vpm167.stdout:3506: done (5 left) 2014-08-08T10:43:51.422 INFO:tasks.rados.rados.0.vpm167.stdout:3508: done (4 left) 2014-08-08T10:43:51.422 INFO:tasks.rados.rados.0.vpm167.stdout:3511: done (3 left) 2014-08-08T10:43:51.422 INFO:tasks.rados.rados.0.vpm167.stdout:3512: done (2 left) 2014-08-08T10:43:51.423 INFO:tasks.rados.rados.0.vpm167.stdout:3513: done (1 left) 2014-08-08T10:43:51.423 INFO:tasks.rados.rados.0.vpm167.stdout:3514: done (0 left) 2014-08-08T10:43:51.423 INFO:tasks.rados.rados.0.vpm167.stdout:3515: write oid 16 current snap is 461 2014-08-08T10:43:51.424 INFO:tasks.rados.rados.0.vpm167.stdout:3515: seq_num 1322 ranges {427604=553981,1394045=472067,2473086=754732,3627603=1} 2014-08-08T10:43:51.452 INFO:tasks.rados.rados.0.vpm167.stdout:3515: writing vpm1675861-16 from 427604 to 981585 tid 1 2014-08-08T10:43:51.484 INFO:tasks.rados.rados.0.vpm167.stdout:3515: writing vpm1675861-16 from 1394045 to 1866112 tid 2 2014-08-08T10:43:51.531 INFO:tasks.rados.rados.0.vpm167.stdout:3515: writing vpm1675861-16 from 2473086 to 3227818 tid 3 2014-08-08T10:43:51.538 INFO:tasks.rados.rados.0.vpm167.stdout:3515: writing vpm1675861-16 from 3627603 to 3627604 tid 4 2014-08-08T10:43:51.540 INFO:tasks.rados.rados.0.vpm167.stdout:3516: read oid 5 snap -1 2014-08-08T10:43:51.540 INFO:tasks.rados.rados.0.vpm167.stdout:3517: write oid 31 current snap is 461 2014-08-08T10:43:51.541 INFO:tasks.rados.rados.0.vpm167.stdout:3517: seq_num 1323 ranges {523858=677209,1658577=542262,2708359=615499} 2014-08-08T10:43:51.580 INFO:tasks.rados.rados.0.vpm167.stdout:3517: writing vpm1675861-31 from 523858 to 1201067 tid 1 2014-08-08T10:43:51.627 INFO:tasks.rados.rados.0.vpm167.stdout:3517: writing vpm1675861-31 from 1658577 to 2200839 tid 2 2014-08-08T10:43:51.660 INFO:tasks.rados.rados.0.vpm167.stdout:3517: writing vpm1675861-31 from 2708359 to 3323858 tid 3 2014-08-08T10:43:51.662 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 1 to vpm1675861-16 2014-08-08T10:43:51.662 INFO:tasks.rados.rados.0.vpm167.stdout:3516: expect (ObjNum 1005 snap 342 seq_num 1005) 2014-08-08T10:43:53.260 INFO:tasks.rados.rados.0.vpm167.stdout:3516: done (2 left) 2014-08-08T10:43:53.261 INFO:tasks.rados.rados.0.vpm167.stdout:3518: read oid 12 snap -1 2014-08-08T10:43:53.261 INFO:tasks.rados.rados.0.vpm167.stdout:3519: write oid 50 current snap is 461 2014-08-08T10:43:53.261 INFO:tasks.rados.rados.0.vpm167.stdout:3519: seq_num 1324 ranges {619088=782262,1904934=647530,3019087=1} 2014-08-08T10:43:53.302 INFO:tasks.rados.rados.0.vpm167.stdout:3519: writing vpm1675861-50 from 619088 to 1401350 tid 1 2014-08-08T10:43:53.332 INFO:tasks.rados.rados.0.vpm167.stdout:3519: writing vpm1675861-50 from 1904934 to 2552464 tid 2 2014-08-08T10:43:53.337 INFO:tasks.rados.rados.0.vpm167.stdout:3519: writing vpm1675861-50 from 3019087 to 3019088 tid 3 2014-08-08T10:43:53.338 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 2 to vpm1675861-16 2014-08-08T10:43:53.338 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 3 to vpm1675861-16 2014-08-08T10:43:53.339 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 4 to vpm1675861-16 2014-08-08T10:43:53.339 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 6 to vpm1675861-16 2014-08-08T10:43:53.339 INFO:tasks.rados.rados.0.vpm167.stdout:3517: finishing write tid 1 to vpm1675861-31 2014-08-08T10:43:53.339 INFO:tasks.rados.rados.0.vpm167.stdout:3517: finishing write tid 2 to vpm1675861-31 2014-08-08T10:43:53.340 INFO:tasks.rados.rados.0.vpm167.stdout:3517: finishing write tid 3 to vpm1675861-31 2014-08-08T10:43:53.340 INFO:tasks.rados.rados.0.vpm167.stdout:3517: finishing write tid 5 to vpm1675861-31 2014-08-08T10:43:53.340 INFO:tasks.rados.rados.0.vpm167.stdout:3515: finishing write tid 7 to vpm1675861-16 2014-08-08T10:43:53.341 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1570 (ObjNum 1322 snap 461 seq_num 1322) dirty exists 2014-08-08T10:43:53.341 INFO:tasks.rados.rados.0.vpm167.stdout:3517: finishing write tid 6 to vpm1675861-31 2014-08-08T10:43:53.341 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1399 (ObjNum 1323 snap 461 seq_num 1323) dirty exists 2014-08-08T10:43:53.341 INFO:tasks.rados.rados.0.vpm167.stdout:3518: expect (ObjNum 1301 snap 456 seq_num 1301) 2014-08-08T10:43:55.289 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:43:55.289 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:43:55.290 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-08T10:43:55.290 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T10:43:55.290 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T10:43:55.290 INFO:teuthology.orchestra.run.vpm166:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-08T10:43:55.490 INFO:tasks.rados.rados.0.vpm167.stdout:3515: done (3 left) 2014-08-08T10:43:55.490 INFO:tasks.rados.rados.0.vpm167.stdout:3517: done (2 left) 2014-08-08T10:43:55.490 INFO:tasks.rados.rados.0.vpm167.stdout:3518: done (1 left) 2014-08-08T10:43:55.492 INFO:tasks.rados.rados.0.vpm167.stdout:3520: snap_create 2014-08-08T10:43:55.492 INFO:tasks.rados.rados.0.vpm167.stdout:3519: finishing write tid 1 to vpm1675861-50 2014-08-08T10:43:55.492 INFO:tasks.rados.rados.0.vpm167.stdout:3519: finishing write tid 2 to vpm1675861-50 2014-08-08T10:43:55.493 INFO:tasks.rados.rados.0.vpm167.stdout:3519: finishing write tid 3 to vpm1675861-50 2014-08-08T10:43:55.493 INFO:tasks.rados.rados.0.vpm167.stdout:3519: finishing write tid 5 to vpm1675861-50 2014-08-08T10:43:55.493 INFO:tasks.rados.rados.0.vpm167.stdout:3519: finishing write tid 6 to vpm1675861-50 2014-08-08T10:43:55.493 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 50 v 818 (ObjNum 1324 snap 461 seq_num 1324) dirty exists 2014-08-08T10:43:55.940 INFO:tasks.rados.rados.0.vpm167.stdout:3519: done (1 left) 2014-08-08T10:43:55.941 INFO:tasks.rados.rados.0.vpm167.stdout:3520: done (0 left) 2014-08-08T10:43:55.941 INFO:tasks.rados.rados.0.vpm167.stdout:3521: write oid 33 current snap is 462 2014-08-08T10:43:55.941 INFO:tasks.rados.rados.0.vpm167.stdout:3521: seq_num 1325 ranges {412493=400000} 2014-08-08T10:43:55.958 INFO:tasks.rados.rados.0.vpm167.stdout:3521: writing vpm1675861-33 from 412493 to 812493 tid 1 2014-08-08T10:43:55.962 INFO:tasks.rados.rados.0.vpm167.stdout:3522: delete oid 20 current snap is 462 2014-08-08T10:43:56.012 INFO:tasks.rados.rados.0.vpm167.stdout:3522: done (1 left) 2014-08-08T10:43:56.012 INFO:tasks.rados.rados.0.vpm167.stdout:3523: rollback oid 14 current snap is 462 2014-08-08T10:43:56.012 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 450 2014-08-08T10:43:56.013 INFO:tasks.rados.rados.0.vpm167.stdout:3524: delete oid 31 current snap is 462 2014-08-08T10:43:56.020 INFO:tasks.rados.rados.0.vpm167.stdout:3521: finishing write tid 1 to vpm1675861-33 2014-08-08T10:43:56.035 INFO:tasks.rados.rados.0.vpm167.stdout:3521: finishing write tid 3 to vpm1675861-33 2014-08-08T10:43:56.035 INFO:tasks.rados.rados.0.vpm167.stdout:3521: finishing write tid 4 to vpm1675861-33 2014-08-08T10:43:56.035 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1929 (ObjNum 1325 snap 462 seq_num 1325) dirty exists 2014-08-08T10:43:56.095 INFO:tasks.rados.rados.0.vpm167.stdout:3521: done (2 left) 2014-08-08T10:43:56.095 INFO:tasks.rados.rados.0.vpm167.stdout:3524: done (1 left) 2014-08-08T10:43:56.096 INFO:tasks.rados.rados.0.vpm167.stdout:3525: read oid 28 snap -1 2014-08-08T10:43:56.096 INFO:tasks.rados.rados.0.vpm167.stdout:3526: rollback oid 4 current snap is 462 2014-08-08T10:43:56.097 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 4 to 434 2014-08-08T10:43:56.097 INFO:tasks.rados.rados.0.vpm167.stdout:3527: read oid 36 snap -1 2014-08-08T10:43:56.097 INFO:tasks.rados.rados.0.vpm167.stdout:3528: delete oid 23 current snap is 462 2014-08-08T10:43:56.128 INFO:tasks.rados.rados.0.vpm167.stdout:3525: expect (ObjNum 1315 snap 459 seq_num 1315) 2014-08-08T10:43:57.463 INFO:tasks.rados.rados.0.vpm167.stdout:3525: done (4 left) 2014-08-08T10:43:57.464 INFO:tasks.rados.rados.0.vpm167.stdout:3527: done (3 left) 2014-08-08T10:43:57.465 INFO:tasks.rados.rados.0.vpm167.stdout:3528: done (2 left) 2014-08-08T10:43:57.465 INFO:tasks.rados.rados.0.vpm167.stdout:3529: rollback oid 48 current snap is 462 2014-08-08T10:43:57.465 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 48 to 450 2014-08-08T10:43:57.466 INFO:tasks.rados.rados.0.vpm167.stdout:3530: read oid 12 snap -1 2014-08-08T10:43:57.466 INFO:tasks.rados.rados.0.vpm167.stdout:3531: rollback oid 30 current snap is 462 2014-08-08T10:43:57.466 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 30 to 432 2014-08-08T10:43:57.467 INFO:tasks.rados.rados.0.vpm167.stdout:3532: read oid 8 snap -1 2014-08-08T10:43:57.467 INFO:tasks.rados.rados.0.vpm167.stdout:3533: snap_create 2014-08-08T10:43:57.468 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 4 v 1308 (ObjNum 1221 snap 427 seq_num 1221) dirty exists 2014-08-08T10:43:57.468 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2212 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:43:57.499 INFO:tasks.rados.rados.0.vpm167.stdout:3530: expect (ObjNum 1301 snap 456 seq_num 1301) 2014-08-08T10:43:59.526 INFO:tasks.rados.rados.0.vpm167.stdout:3523: done (6 left) 2014-08-08T10:43:59.526 INFO:tasks.rados.rados.0.vpm167.stdout:3526: done (5 left) 2014-08-08T10:43:59.526 INFO:tasks.rados.rados.0.vpm167.stdout:3530: done (4 left) 2014-08-08T10:43:59.527 INFO:tasks.rados.rados.0.vpm167.stdout:3533: done (3 left) 2014-08-08T10:43:59.527 INFO:tasks.rados.rados.0.vpm167.stdout:3534: snap_create 2014-08-08T10:43:59.527 INFO:tasks.rados.rados.0.vpm167.stdout:3532: expect (ObjNum 1063 snap 368 seq_num 1063) 2014-08-08T10:44:00.379 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:00.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:00.380 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2014-08-08T10:44:00.380 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 1' 2014-08-08T10:44:00.735 INFO:tasks.rados.rados.0.vpm167.stdout:3532: done (3 left) 2014-08-08T10:44:00.735 INFO:tasks.rados.rados.0.vpm167.stdout:3534: done (2 left) 2014-08-08T10:44:00.735 INFO:tasks.rados.rados.0.vpm167.stdout:3535: snap_create 2014-08-08T10:44:00.737 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 30 v 1310 (ObjNum 1052 snap 362 seq_num 1052) dirty exists 2014-08-08T10:44:00.737 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 48 v 1931 (ObjNum 1269 snap 440 seq_num 1269) dirty exists 2014-08-08T10:44:01.582 INFO:teuthology.orchestra.run.vpm166.stderr:set osd.0 primary-affinity to 1 (8655362) 2014-08-08T10:44:01.683 INFO:tasks.rados.rados.0.vpm167.stdout:3529: done (2 left) 2014-08-08T10:44:01.683 INFO:tasks.rados.rados.0.vpm167.stdout:3531: done (1 left) 2014-08-08T10:44:01.684 INFO:tasks.rados.rados.0.vpm167.stdout:3535: done (0 left) 2014-08-08T10:44:01.684 INFO:tasks.rados.rados.0.vpm167.stdout:3536: write oid 17 current snap is 465 2014-08-08T10:44:01.684 INFO:tasks.rados.rados.0.vpm167.stdout:3536: seq_num 1329 ranges {608964=580227,1701577=507387} 2014-08-08T10:44:01.708 INFO:tasks.rados.rados.0.vpm167.stdout:3536: writing vpm1675861-17 from 608964 to 1189191 tid 1 2014-08-08T10:44:01.738 INFO:tasks.rados.rados.0.vpm167.stdout:3536: writing vpm1675861-17 from 1701577 to 2208964 tid 2 2014-08-08T10:44:01.740 INFO:tasks.rados.rados.0.vpm167.stdout:3536: finishing write tid 1 to vpm1675861-17 2014-08-08T10:44:01.740 INFO:tasks.rados.rados.0.vpm167.stdout:3537: write oid 31 current snap is 465 2014-08-08T10:44:01.740 INFO:tasks.rados.rados.0.vpm167.stdout:3537: seq_num 1330 ranges {420546=552032,1647390=571549,2420545=1} 2014-08-08T10:44:01.761 INFO:tasks.rados.rados.0.vpm167.stdout:3537: writing vpm1675861-31 from 420546 to 972578 tid 1 2014-08-08T10:44:01.809 INFO:tasks.rados.rados.0.vpm167.stdout:3537: writing vpm1675861-31 from 1647390 to 2218939 tid 2 2014-08-08T10:44:01.813 INFO:tasks.rados.rados.0.vpm167.stdout:3537: writing vpm1675861-31 from 2420545 to 2420546 tid 3 2014-08-08T10:44:01.814 INFO:tasks.rados.rados.0.vpm167.stdout:3536: finishing write tid 2 to vpm1675861-17 2014-08-08T10:44:01.814 INFO:tasks.rados.rados.0.vpm167.stdout:3536: finishing write tid 4 to vpm1675861-17 2014-08-08T10:44:01.814 INFO:tasks.rados.rados.0.vpm167.stdout:3536: finishing write tid 5 to vpm1675861-17 2014-08-08T10:44:01.814 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2321 (ObjNum 1329 snap 465 seq_num 1329) dirty exists 2014-08-08T10:44:01.814 INFO:tasks.rados.rados.0.vpm167.stdout:3537: finishing write tid 1 to vpm1675861-31 2014-08-08T10:44:01.815 INFO:tasks.rados.rados.0.vpm167.stdout:3536: done (1 left) 2014-08-08T10:44:01.815 INFO:tasks.rados.rados.0.vpm167.stdout:3538: delete oid 15 current snap is 465 2014-08-08T10:44:01.817 INFO:tasks.rados.rados.0.vpm167.stdout:3538: done (1 left) 2014-08-08T10:44:01.817 INFO:tasks.rados.rados.0.vpm167.stdout:3539: snap_create 2014-08-08T10:44:01.843 INFO:tasks.rados.rados.0.vpm167.stdout:3537: finishing write tid 2 to vpm1675861-31 2014-08-08T10:44:01.844 INFO:tasks.rados.rados.0.vpm167.stdout:3537: finishing write tid 3 to vpm1675861-31 2014-08-08T10:44:01.844 INFO:tasks.rados.rados.0.vpm167.stdout:3537: finishing write tid 5 to vpm1675861-31 2014-08-08T10:44:01.844 INFO:tasks.rados.rados.0.vpm167.stdout:3537: finishing write tid 6 to vpm1675861-31 2014-08-08T10:44:01.845 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1407 (ObjNum 1330 snap 465 seq_num 1330) dirty exists 2014-08-08T10:44:02.820 INFO:tasks.rados.rados.0.vpm167.stdout:3537: done (1 left) 2014-08-08T10:44:02.820 INFO:tasks.rados.rados.0.vpm167.stdout:3539: done (0 left) 2014-08-08T10:44:02.820 INFO:tasks.rados.rados.0.vpm167.stdout:3540: read oid 37 snap -1 2014-08-08T10:44:02.821 INFO:tasks.rados.rados.0.vpm167.stdout:3541: read oid 47 snap -1 2014-08-08T10:44:02.821 INFO:tasks.rados.rados.0.vpm167.stdout:3542: read oid 23 snap -1 2014-08-08T10:44:02.822 INFO:tasks.rados.rados.0.vpm167.stdout:3543: rollback oid 14 current snap is 466 2014-08-08T10:44:02.822 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 441 2014-08-08T10:44:02.823 INFO:tasks.rados.rados.0.vpm167.stdout:3544: read oid 15 snap -1 2014-08-08T10:44:02.823 INFO:tasks.rados.rados.0.vpm167.stdout:3545: rollback oid 11 current snap is 466 2014-08-08T10:44:02.824 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 11 to 445 2014-08-08T10:44:02.824 INFO:tasks.rados.rados.0.vpm167.stdout:3546: read oid 22 snap 415 2014-08-08T10:44:02.825 INFO:tasks.rados.rados.0.vpm167.stdout:3547: snap_create 2014-08-08T10:44:02.887 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 1240 (ObjNum 1141 snap 404 seq_num 1141) dirty exists 2014-08-08T10:44:03.108 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2214 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:44:04.042 INFO:tasks.rados.rados.0.vpm167.stdout:3540: done (7 left) 2014-08-08T10:44:04.042 INFO:tasks.rados.rados.0.vpm167.stdout:3541: done (6 left) 2014-08-08T10:44:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3542: done (5 left) 2014-08-08T10:44:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3543: done (4 left) 2014-08-08T10:44:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3544: done (3 left) 2014-08-08T10:44:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3545: done (2 left) 2014-08-08T10:44:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3546: done (1 left) 2014-08-08T10:44:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3547: done (0 left) 2014-08-08T10:44:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3548: read oid 36 snap -1 2014-08-08T10:44:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3549: snap_create 2014-08-08T10:44:05.246 INFO:tasks.rados.rados.0.vpm167.stdout:3548: done (1 left) 2014-08-08T10:44:05.246 INFO:tasks.rados.rados.0.vpm167.stdout:3549: done (0 left) 2014-08-08T10:44:05.247 INFO:tasks.rados.rados.0.vpm167.stdout:3550: read oid 26 snap -1 2014-08-08T10:44:05.247 INFO:tasks.rados.rados.0.vpm167.stdout:3551: snap_create 2014-08-08T10:44:05.351 INFO:tasks.rados.rados.0.vpm167.stdout:3550: expect (ObjNum 1300 snap 456 seq_num 1300) 2014-08-08T10:44:06.234 INFO:tasks.rados.rados.0.vpm167.stdout:3550: done (1 left) 2014-08-08T10:44:06.234 INFO:tasks.rados.rados.0.vpm167.stdout:3551: done (0 left) 2014-08-08T10:44:06.234 INFO:tasks.rados.rados.0.vpm167.stdout:3552: write oid 37 current snap is 469 2014-08-08T10:44:06.235 INFO:tasks.rados.rados.0.vpm167.stdout:3552: seq_num 1332 ranges {709182=648088,2088134=710915,3209820=588525,3909181=1} 2014-08-08T10:44:06.257 INFO:tasks.rados.rados.0.vpm167.stdout:3552: writing vpm1675861-37 from 709182 to 1357270 tid 1 2014-08-08T10:44:06.297 INFO:tasks.rados.rados.0.vpm167.stdout:3552: writing vpm1675861-37 from 2088134 to 2799049 tid 2 2014-08-08T10:44:06.321 INFO:tasks.rados.rados.0.vpm167.stdout:3552: writing vpm1675861-37 from 3209820 to 3798345 tid 3 2014-08-08T10:44:06.324 INFO:tasks.rados.rados.0.vpm167.stdout:3552: writing vpm1675861-37 from 3909181 to 3909182 tid 4 2014-08-08T10:44:06.325 INFO:tasks.rados.rados.0.vpm167.stdout:3553: write oid 21 current snap is 469 2014-08-08T10:44:06.325 INFO:tasks.rados.rados.0.vpm167.stdout:3553: seq_num 1333 ranges {405435=654965,1837339=416182,2682214=474292,3605434=1} 2014-08-08T10:44:06.350 INFO:tasks.rados.rados.0.vpm167.stdout:3553: writing vpm1675861-21 from 405435 to 1060400 tid 1 2014-08-08T10:44:06.377 INFO:tasks.rados.rados.0.vpm167.stdout:3553: writing vpm1675861-21 from 1837339 to 2253521 tid 2 2014-08-08T10:44:06.398 INFO:tasks.rados.rados.0.vpm167.stdout:3553: writing vpm1675861-21 from 2682214 to 3156506 tid 3 2014-08-08T10:44:06.403 INFO:tasks.rados.rados.0.vpm167.stdout:3553: writing vpm1675861-21 from 3605434 to 3605435 tid 4 2014-08-08T10:44:06.404 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 1 to vpm1675861-37 2014-08-08T10:44:06.404 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 2 to vpm1675861-37 2014-08-08T10:44:06.404 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 1 to vpm1675861-21 2014-08-08T10:44:06.405 INFO:tasks.rados.rados.0.vpm167.stdout:3554: write oid 14 current snap is 469 2014-08-08T10:44:06.405 INFO:tasks.rados.rados.0.vpm167.stdout:3554: seq_num 1334 ranges {349240=1} 2014-08-08T10:44:06.407 INFO:tasks.rados.rados.0.vpm167.stdout:3554: writing vpm1675861-14 from 349240 to 349241 tid 1 2014-08-08T10:44:06.408 INFO:tasks.rados.rados.0.vpm167.stdout:3555: delete oid 28 current snap is 469 2014-08-08T10:44:06.452 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 3 to vpm1675861-37 2014-08-08T10:44:06.452 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 4 to vpm1675861-37 2014-08-08T10:44:06.458 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 2 to vpm1675861-21 2014-08-08T10:44:06.461 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 3 to vpm1675861-21 2014-08-08T10:44:06.469 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 6 to vpm1675861-37 2014-08-08T10:44:06.470 INFO:tasks.rados.rados.0.vpm167.stdout:3552: finishing write tid 7 to vpm1675861-37 2014-08-08T10:44:06.470 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 37 v 2220 (ObjNum 1332 snap 469 seq_num 1332) dirty exists 2014-08-08T10:44:06.586 INFO:tasks.rados.rados.0.vpm167.stdout:3554: finishing write tid 1 to vpm1675861-14 2014-08-08T10:44:06.594 INFO:tasks.rados.rados.0.vpm167.stdout:3554: finishing write tid 3 to vpm1675861-14 2014-08-08T10:44:06.595 INFO:tasks.rados.rados.0.vpm167.stdout:3554: finishing write tid 4 to vpm1675861-14 2014-08-08T10:44:06.595 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2223 (ObjNum 1334 snap 469 seq_num 1334) dirty exists 2014-08-08T10:44:06.600 INFO:tasks.rados.rados.0.vpm167.stdout:3552: done (3 left) 2014-08-08T10:44:06.600 INFO:tasks.rados.rados.0.vpm167.stdout:3554: done (2 left) 2014-08-08T10:44:06.600 INFO:tasks.rados.rados.0.vpm167.stdout:3555: done (1 left) 2014-08-08T10:44:06.601 INFO:tasks.rados.rados.0.vpm167.stdout:3556: read oid 7 snap 465 2014-08-08T10:44:06.601 INFO:tasks.rados.rados.0.vpm167.stdout:3557: write oid 42 current snap is 469 2014-08-08T10:44:06.601 INFO:tasks.rados.rados.0.vpm167.stdout:3557: seq_num 1336 ranges {505652=571402,1305651=1} 2014-08-08T10:44:06.621 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:06.621 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:06.621 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:06.638 INFO:tasks.rados.rados.0.vpm167.stdout:3557: writing vpm1675861-42 from 505652 to 1077054 tid 1 2014-08-08T10:44:06.641 INFO:tasks.rados.rados.0.vpm167.stdout:3557: writing vpm1675861-42 from 1305651 to 1305652 tid 2 2014-08-08T10:44:06.642 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 4 to vpm1675861-21 2014-08-08T10:44:06.642 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 6 to vpm1675861-21 2014-08-08T10:44:06.642 INFO:tasks.rados.rados.0.vpm167.stdout:3553: finishing write tid 7 to vpm1675861-21 2014-08-08T10:44:06.643 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 21 v 460 (ObjNum 1333 snap 469 seq_num 1333) dirty exists 2014-08-08T10:44:06.643 INFO:tasks.rados.rados.0.vpm167.stdout:3556: expect (ObjNum 1279 snap 441 seq_num 1279) 2014-08-08T10:44:07.446 INFO:tasks.rados.rados.0.vpm167.stdout:3553: done (2 left) 2014-08-08T10:44:07.447 INFO:tasks.rados.rados.0.vpm167.stdout:3556: done (1 left) 2014-08-08T10:44:07.447 INFO:tasks.rados.rados.0.vpm167.stdout:3558: write oid 16 current snap is 469 2014-08-08T10:44:07.447 INFO:tasks.rados.rados.0.vpm167.stdout:3558: seq_num 1337 ranges {450482=579302,1672299=378183} 2014-08-08T10:44:07.480 INFO:tasks.rados.rados.0.vpm167.stdout:3558: writing vpm1675861-16 from 450482 to 1029784 tid 1 2014-08-08T10:44:07.503 INFO:tasks.rados.rados.0.vpm167.stdout:3558: writing vpm1675861-16 from 1672299 to 2050482 tid 2 2014-08-08T10:44:07.505 INFO:tasks.rados.rados.0.vpm167.stdout:3557: finishing write tid 1 to vpm1675861-42 2014-08-08T10:44:07.505 INFO:tasks.rados.rados.0.vpm167.stdout:3557: finishing write tid 2 to vpm1675861-42 2014-08-08T10:44:07.505 INFO:tasks.rados.rados.0.vpm167.stdout:3557: finishing write tid 4 to vpm1675861-42 2014-08-08T10:44:07.506 INFO:tasks.rados.rados.0.vpm167.stdout:3557: finishing write tid 5 to vpm1675861-42 2014-08-08T10:44:07.507 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 2227 (ObjNum 1336 snap 469 seq_num 1336) dirty exists 2014-08-08T10:44:07.507 INFO:tasks.rados.rados.0.vpm167.stdout:3557: done (1 left) 2014-08-08T10:44:07.508 INFO:tasks.rados.rados.0.vpm167.stdout:3559: write oid 41 current snap is 469 2014-08-08T10:44:07.508 INFO:tasks.rados.rados.0.vpm167.stdout:3559: seq_num 1338 ranges {794288=552130,1883583=670006,2794287=1} 2014-08-08T10:44:07.543 INFO:tasks.rados.rados.0.vpm167.stdout:3559: writing vpm1675861-41 from 794288 to 1346418 tid 1 2014-08-08T10:44:07.588 INFO:tasks.rados.rados.0.vpm167.stdout:3559: writing vpm1675861-41 from 1883583 to 2553589 tid 2 2014-08-08T10:44:07.593 INFO:tasks.rados.rados.0.vpm167.stdout:3559: writing vpm1675861-41 from 2794287 to 2794288 tid 3 2014-08-08T10:44:07.595 INFO:tasks.rados.rados.0.vpm167.stdout:3559: finishing write tid 1 to vpm1675861-41 2014-08-08T10:44:07.595 INFO:tasks.rados.rados.0.vpm167.stdout:3560: read oid 5 snap -1 2014-08-08T10:44:07.596 INFO:tasks.rados.rados.0.vpm167.stdout:3561: snap_remove snap 450 2014-08-08T10:44:07.620 INFO:tasks.rados.rados.0.vpm167.stdout:3560: expect (ObjNum 1005 snap 342 seq_num 1005) 2014-08-08T10:44:10.187 INFO:tasks.rados.rados.0.vpm167.stdout:3560: done (3 left) 2014-08-08T10:44:10.187 INFO:tasks.rados.rados.0.vpm167.stdout:3561: done (2 left) 2014-08-08T10:44:10.187 INFO:tasks.rados.rados.0.vpm167.stdout:3562: delete oid 19 current snap is 469 2014-08-08T10:44:10.188 INFO:tasks.rados.rados.0.vpm167.stdout:3559: finishing write tid 2 to vpm1675861-41 2014-08-08T10:44:10.188 INFO:tasks.rados.rados.0.vpm167.stdout:3559: finishing write tid 3 to vpm1675861-41 2014-08-08T10:44:10.189 INFO:tasks.rados.rados.0.vpm167.stdout:3559: finishing write tid 5 to vpm1675861-41 2014-08-08T10:44:10.189 INFO:tasks.rados.rados.0.vpm167.stdout:3559: finishing write tid 6 to vpm1675861-41 2014-08-08T10:44:10.190 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1870 (ObjNum 1338 snap 469 seq_num 1338) dirty exists 2014-08-08T10:44:12.622 INFO:tasks.ceph.osd.3:Stopped 2014-08-08T10:44:17.623 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 5, 4, 2] 2014-08-08T10:44:17.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:17.623 INFO:tasks.thrashosds.thrasher:fixing pg num pool data 2014-08-08T10:44:17.624 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:44:17.919 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:44:17.994 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 54' 2014-08-08T10:44:18.378 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 0 pgp_num to 54 2014-08-08T10:44:23.447 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 5, 4, 2] 2014-08-08T10:44:23.447 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:23.447 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2014-08-08T10:44:23.447 INFO:tasks.ceph.osd.3:Restarting daemon 2014-08-08T10:44:23.448 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-08-08T10:44:23.450 INFO:tasks.ceph.osd.3:Started 2014-08-08T10:44:23.451 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-08-08T10:44:23.582 INFO:teuthology.orchestra.run.vpm165.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2014-08-08T10:44:23.591 INFO:tasks.thrashosds.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2014-08-08T10:44:23.743 INFO:tasks.ceph.osd.3.vpm165.stdout:starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-08-08T10:44:23.849 INFO:tasks.ceph.osd.3.vpm165.stderr:2014-08-08 17:44:23.847571 7f80acd6d8c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-08-08T10:44:27.511 INFO:tasks.rados.rados.0.vpm167.stdout:3558: finishing write tid 1 to vpm1675861-16 2014-08-08T10:44:27.511 INFO:tasks.rados.rados.0.vpm167.stdout:3558: finishing write tid 2 to vpm1675861-16 2014-08-08T10:44:27.511 INFO:tasks.rados.rados.0.vpm167.stdout:3558: finishing write tid 4 to vpm1675861-16 2014-08-08T10:44:27.512 INFO:tasks.rados.rados.0.vpm167.stdout:3558: finishing write tid 5 to vpm1675861-16 2014-08-08T10:44:27.512 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 16 v 1574 (ObjNum 1337 snap 469 seq_num 1337) dirty exists 2014-08-08T10:44:28.591 INFO:teuthology.orchestra.run.vpm165:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-08-08T10:44:28.812 INFO:tasks.rados.rados.0.vpm167.stdout:3558: done (2 left) 2014-08-08T10:44:28.812 INFO:tasks.rados.rados.0.vpm167.stdout:3559: done (1 left) 2014-08-08T10:44:28.812 INFO:tasks.rados.rados.0.vpm167.stdout:3562: done (0 left) 2014-08-08T10:44:28.813 INFO:tasks.rados.rados.0.vpm167.stdout:3563: read oid 27 snap -1 2014-08-08T10:44:28.813 INFO:tasks.rados.rados.0.vpm167.stdout:3564: delete oid 50 current snap is 469 2014-08-08T10:44:28.833 INFO:tasks.rados.rados.0.vpm167.stdout:3563: expect (ObjNum 1296 snap 453 seq_num 1296) 2014-08-08T10:44:30.397 INFO:tasks.rados.rados.0.vpm167.stdout:3563: done (1 left) 2014-08-08T10:44:30.398 INFO:tasks.rados.rados.0.vpm167.stdout:3564: done (0 left) 2014-08-08T10:44:30.398 INFO:tasks.rados.rados.0.vpm167.stdout:3565: write oid 15 current snap is 469 2014-08-08T10:44:30.398 INFO:tasks.rados.rados.0.vpm167.stdout:3565: seq_num 1341 ranges {494504=1} 2014-08-08T10:44:30.401 INFO:tasks.rados.rados.0.vpm167.stdout:3565: writing vpm1675861-15 from 494504 to 494505 tid 1 2014-08-08T10:44:30.402 INFO:tasks.rados.rados.0.vpm167.stdout:3566: read oid 48 snap -1 2014-08-08T10:44:30.403 INFO:tasks.rados.rados.0.vpm167.stdout:3567: write oid 28 current snap is 469 2014-08-08T10:44:30.403 INFO:tasks.rados.rados.0.vpm167.stdout:3567: seq_num 1342 ranges {190758=1} 2014-08-08T10:44:30.405 INFO:tasks.rados.rados.0.vpm167.stdout:3567: writing vpm1675861-28 from 190758 to 190759 tid 1 2014-08-08T10:44:30.406 INFO:tasks.rados.rados.0.vpm167.stdout:3568: rollback oid 46 current snap is 469 2014-08-08T10:44:30.406 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 46 to 464 2014-08-08T10:44:30.407 INFO:tasks.rados.rados.0.vpm167.stdout:3569: read oid 30 snap -1 2014-08-08T10:44:30.407 INFO:tasks.rados.rados.0.vpm167.stdout:3570: delete oid 41 current snap is 469 2014-08-08T10:44:30.489 INFO:tasks.rados.rados.0.vpm167.stdout:3570: done (5 left) 2014-08-08T10:44:30.489 INFO:tasks.rados.rados.0.vpm167.stdout:3571: read oid 26 snap -1 2014-08-08T10:44:30.489 INFO:tasks.rados.rados.0.vpm167.stdout:3572: snap_create 2014-08-08T10:44:30.494 INFO:tasks.rados.rados.0.vpm167.stdout:3571: expect (ObjNum 1300 snap 456 seq_num 1300) 2014-08-08T10:44:30.927 INFO:tasks.rados.rados.0.vpm167.stdout:3566: expect (ObjNum 1269 snap 440 seq_num 1269) 2014-08-08T10:44:31.224 INFO:tasks.rados.rados.0.vpm167.stdout:3569: expect (ObjNum 1052 snap 362 seq_num 1052) 2014-08-08T10:44:32.046 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 46 v 1941 (ObjNum 1060 snap 364 seq_num 1060) dirty exists 2014-08-08T10:44:32.046 INFO:tasks.rados.rados.0.vpm167.stdout:3565: finishing write tid 1 to vpm1675861-15 2014-08-08T10:44:32.047 INFO:tasks.rados.rados.0.vpm167.stdout:3565: finishing write tid 3 to vpm1675861-15 2014-08-08T10:44:32.047 INFO:tasks.rados.rados.0.vpm167.stdout:3565: finishing write tid 4 to vpm1675861-15 2014-08-08T10:44:32.047 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 15 v 1944 (ObjNum 1341 snap 469 seq_num 1341) dirty exists 2014-08-08T10:44:32.048 INFO:tasks.rados.rados.0.vpm167.stdout:3567: finishing write tid 1 to vpm1675861-28 2014-08-08T10:44:32.048 INFO:tasks.rados.rados.0.vpm167.stdout:3567: finishing write tid 3 to vpm1675861-28 2014-08-08T10:44:32.048 INFO:tasks.rados.rados.0.vpm167.stdout:3567: finishing write tid 4 to vpm1675861-28 2014-08-08T10:44:32.048 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 28 v 831 (ObjNum 1342 snap 469 seq_num 1342) dirty exists 2014-08-08T10:44:32.220 INFO:tasks.rados.rados.0.vpm167.stdout:3565: done (6 left) 2014-08-08T10:44:32.221 INFO:tasks.rados.rados.0.vpm167.stdout:3566: done (5 left) 2014-08-08T10:44:32.221 INFO:tasks.rados.rados.0.vpm167.stdout:3567: done (4 left) 2014-08-08T10:44:32.221 INFO:tasks.rados.rados.0.vpm167.stdout:3568: done (3 left) 2014-08-08T10:44:32.221 INFO:tasks.rados.rados.0.vpm167.stdout:3569: done (2 left) 2014-08-08T10:44:32.222 INFO:tasks.rados.rados.0.vpm167.stdout:3571: done (1 left) 2014-08-08T10:44:32.222 INFO:tasks.rados.rados.0.vpm167.stdout:3572: done (0 left) 2014-08-08T10:44:32.222 INFO:tasks.rados.rados.0.vpm167.stdout:3573: snap_remove snap 441 2014-08-08T10:44:33.288 INFO:tasks.rados.rados.0.vpm167.stdout:3573: done (0 left) 2014-08-08T10:44:33.288 INFO:tasks.rados.rados.0.vpm167.stdout:3574: rollback oid 11 current snap is 470 2014-08-08T10:44:33.289 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 11 to 344 2014-08-08T10:44:33.289 INFO:tasks.rados.rados.0.vpm167.stdout:3575: write oid 41 current snap is 470 2014-08-08T10:44:33.289 INFO:tasks.rados.rados.0.vpm167.stdout:3575: seq_num 1344 ranges {747170=400000} 2014-08-08T10:44:33.315 INFO:tasks.rados.rados.0.vpm167.stdout:3575: writing vpm1675861-41 from 747170 to 1147170 tid 1 2014-08-08T10:44:33.318 INFO:tasks.rados.rados.0.vpm167.stdout:3576: snap_create 2014-08-08T10:44:33.386 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 11 v 1246 (ObjNum 1001 snap 342 seq_num 1001) dirty exists 2014-08-08T10:44:33.614 INFO:tasks.rados.rados.0.vpm167.stdout:3575: finishing write tid 1 to vpm1675861-41 2014-08-08T10:44:33.615 INFO:tasks.rados.rados.0.vpm167.stdout:3575: finishing write tid 3 to vpm1675861-41 2014-08-08T10:44:33.638 INFO:tasks.rados.rados.0.vpm167.stdout:3575: finishing write tid 4 to vpm1675861-41 2014-08-08T10:44:33.638 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1877 (ObjNum 1344 snap 470 seq_num 1344) dirty exists 2014-08-08T10:44:33.751 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:33.751 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:33.751 INFO:tasks.thrashosds.thrasher:Growing pool data 2014-08-08T10:44:33.752 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:44:33.963 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:44:34.022 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T10:44:34.022 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 64' 2014-08-08T10:44:34.429 INFO:tasks.rados.rados.0.vpm167.stdout:3574: done (2 left) 2014-08-08T10:44:34.429 INFO:tasks.rados.rados.0.vpm167.stdout:3575: done (1 left) 2014-08-08T10:44:34.429 INFO:tasks.rados.rados.0.vpm167.stdout:3576: done (0 left) 2014-08-08T10:44:34.429 INFO:tasks.rados.rados.0.vpm167.stdout:3577: write oid 10 current snap is 471 2014-08-08T10:44:34.430 INFO:tasks.rados.rados.0.vpm167.stdout:3577: seq_num 1345 ranges {692000=729801,1891999=1} 2014-08-08T10:44:34.458 INFO:tasks.rados.rados.0.vpm167.stdout:3577: writing vpm1675861-10 from 692000 to 1421801 tid 1 2014-08-08T10:44:34.473 INFO:tasks.rados.rados.0.vpm167.stdout:3577: writing vpm1675861-10 from 1891999 to 1892000 tid 2 2014-08-08T10:44:34.474 INFO:tasks.rados.rados.0.vpm167.stdout:3578: read oid 7 snap 408 2014-08-08T10:44:34.475 INFO:tasks.rados.rados.0.vpm167.stdout:3579: delete oid 27 current snap is 471 2014-08-08T10:44:34.657 INFO:tasks.rados.rados.0.vpm167.stdout:3577: finishing write tid 1 to vpm1675861-10 2014-08-08T10:44:35.175 INFO:tasks.rados.rados.0.vpm167.stdout:3577: finishing write tid 2 to vpm1675861-10 2014-08-08T10:44:35.175 INFO:tasks.rados.rados.0.vpm167.stdout:3577: finishing write tid 4 to vpm1675861-10 2014-08-08T10:44:35.196 INFO:tasks.rados.rados.0.vpm167.stdout:3578: expect (ObjNum 1156 snap 407 seq_num 1156) 2014-08-08T10:44:35.554 INFO:teuthology.orchestra.run.vpm166.stderr:set pool 0 pg_num to 64 2014-08-08T10:44:35.711 INFO:tasks.rados.rados.0.vpm167.stdout:3577: finishing write tid 5 to vpm1675861-10 2014-08-08T10:44:35.711 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 10 v 1005 (ObjNum 1345 snap 471 seq_num 1345) dirty exists 2014-08-08T10:44:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:3577: done (2 left) 2014-08-08T10:44:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:3578: done (1 left) 2014-08-08T10:44:37.395 INFO:tasks.rados.rados.0.vpm167.stdout:3579: done (0 left) 2014-08-08T10:44:37.396 INFO:tasks.rados.rados.0.vpm167.stdout:3580: write oid 43 current snap is 471 2014-08-08T10:44:37.396 INFO:tasks.rados.rados.0.vpm167.stdout:3580: seq_num 1347 ranges {731036=673409,1965364=365672} 2014-08-08T10:44:37.425 INFO:tasks.rados.rados.0.vpm167.stdout:3580: writing vpm1675861-43 from 731036 to 1404445 tid 1 2014-08-08T10:44:37.454 INFO:tasks.rados.rados.0.vpm167.stdout:3580: writing vpm1675861-43 from 1965364 to 2331036 tid 2 2014-08-08T10:44:37.456 INFO:tasks.rados.rados.0.vpm167.stdout:3581: read oid 9 snap 456 2014-08-08T10:44:37.458 INFO:tasks.rados.rados.0.vpm167.stdout:3582: read oid 45 snap -1 2014-08-08T10:44:37.458 INFO:tasks.rados.rados.0.vpm167.stdout:3583: read oid 3 snap 415 2014-08-08T10:44:37.458 INFO:tasks.rados.rados.0.vpm167.stdout:3584: read oid 44 snap 440 2014-08-08T10:44:37.462 INFO:tasks.rados.rados.0.vpm167.stdout:3585: snap_remove snap 426 2014-08-08T10:44:37.471 INFO:tasks.rados.rados.0.vpm167.stdout:3581: expect (ObjNum 1266 snap 440 seq_num 1266) 2014-08-08T10:44:37.514 INFO:tasks.rados.rados.0.vpm167.stdout:3582: expect (ObjNum 1308 snap 457 seq_num 1308) 2014-08-08T10:44:39.878 INFO:tasks.rados.rados.0.vpm167.stdout:3581: done (5 left) 2014-08-08T10:44:39.878 INFO:tasks.rados.rados.0.vpm167.stdout:3582: done (4 left) 2014-08-08T10:44:39.878 INFO:tasks.rados.rados.0.vpm167.stdout:3583: done (3 left) 2014-08-08T10:44:39.879 INFO:tasks.rados.rados.0.vpm167.stdout:3585: done (2 left) 2014-08-08T10:44:39.879 INFO:tasks.rados.rados.0.vpm167.stdout:3586: snap_remove snap 470 2014-08-08T10:44:39.879 INFO:tasks.rados.rados.0.vpm167.stdout:3584: expect (ObjNum 1278 snap 440 seq_num 1278) 2014-08-08T10:44:40.560 INFO:tasks.rados.rados.0.vpm167.stdout:3584: done (2 left) 2014-08-08T10:44:40.560 INFO:tasks.rados.rados.0.vpm167.stdout:3586: done (1 left) 2014-08-08T10:44:40.560 INFO:tasks.rados.rados.0.vpm167.stdout:3587: delete oid 18 current snap is 471 2014-08-08T10:44:40.561 INFO:tasks.rados.rados.0.vpm167.stdout:3580: finishing write tid 1 to vpm1675861-43 2014-08-08T10:44:40.561 INFO:tasks.rados.rados.0.vpm167.stdout:3580: finishing write tid 2 to vpm1675861-43 2014-08-08T10:44:40.561 INFO:tasks.rados.rados.0.vpm167.stdout:3580: finishing write tid 4 to vpm1675861-43 2014-08-08T10:44:40.561 INFO:tasks.rados.rados.0.vpm167.stdout:3580: finishing write tid 5 to vpm1675861-43 2014-08-08T10:44:40.562 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 888 (ObjNum 1347 snap 471 seq_num 1347) dirty exists 2014-08-08T10:44:40.586 INFO:tasks.rados.rados.0.vpm167.stdout:3580: done (1 left) 2014-08-08T10:44:40.587 INFO:tasks.rados.rados.0.vpm167.stdout:3587: done (0 left) 2014-08-08T10:44:40.587 INFO:tasks.rados.rados.0.vpm167.stdout:3588: write oid 35 current snap is 471 2014-08-08T10:44:40.587 INFO:tasks.rados.rados.0.vpm167.stdout:3588: seq_num 1349 ranges {336022=1} 2014-08-08T10:44:40.590 INFO:tasks.rados.rados.0.vpm167.stdout:3588: writing vpm1675861-35 from 336022 to 336023 tid 1 2014-08-08T10:44:40.591 INFO:tasks.rados.rados.0.vpm167.stdout:3589: delete oid 6 current snap is 471 2014-08-08T10:44:40.597 INFO:tasks.rados.rados.0.vpm167.stdout:3589: done (1 left) 2014-08-08T10:44:40.597 INFO:tasks.rados.rados.0.vpm167.stdout:3590: snap_create 2014-08-08T10:44:40.609 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 0, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:40.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T10:44:40.610 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 4, 3, 2, 0, 5] 2014-08-08T10:44:40.610 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-08T10:44:40.677 INFO:tasks.rados.rados.0.vpm167.stdout:3588: finishing write tid 1 to vpm1675861-35 2014-08-08T10:44:40.706 INFO:tasks.rados.rados.0.vpm167.stdout:3588: finishing write tid 3 to vpm1675861-35 2014-08-08T10:44:40.744 INFO:tasks.rados.rados.0.vpm167.stdout:3588: finishing write tid 4 to vpm1675861-35 2014-08-08T10:44:40.744 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 35 v 1253 (ObjNum 1349 snap 471 seq_num 1349) dirty exists 2014-08-08T10:44:41.266 INFO:teuthology.orchestra.run.vpm166.stderr:marked out osd.0. 2014-08-08T10:44:41.369 INFO:tasks.rados.rados.0.vpm167.stdout:3588: done (1 left) 2014-08-08T10:44:41.370 INFO:tasks.rados.rados.0.vpm167.stdout:3590: done (0 left) 2014-08-08T10:44:41.370 INFO:tasks.rados.rados.0.vpm167.stdout:3591: snap_remove snap 457 2014-08-08T10:44:42.764 INFO:tasks.rados.rados.0.vpm167.stdout:3591: done (0 left) 2014-08-08T10:44:42.764 INFO:tasks.rados.rados.0.vpm167.stdout:3592: delete oid 38 current snap is 472 2014-08-08T10:44:42.767 INFO:tasks.rados.rados.0.vpm167.stdout:3592: done (0 left) 2014-08-08T10:44:42.768 INFO:tasks.rados.rados.0.vpm167.stdout:3593: snap_create 2014-08-08T10:44:43.861 INFO:tasks.rados.rados.0.vpm167.stdout:3593: done (0 left) 2014-08-08T10:44:43.861 INFO:tasks.rados.rados.0.vpm167.stdout:3594: delete oid 32 current snap is 473 2014-08-08T10:44:44.551 INFO:tasks.rados.rados.0.vpm167.stdout:3594: done (0 left) 2014-08-08T10:44:44.552 INFO:tasks.rados.rados.0.vpm167.stdout:3595: rollback oid 27 current snap is 473 2014-08-08T10:44:44.552 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 27 to 466 2014-08-08T10:44:44.552 INFO:tasks.rados.rados.0.vpm167.stdout:3596: snap_create 2014-08-08T10:44:44.920 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 27 v 2351 (ObjNum 1296 snap 453 seq_num 1296) dirty exists 2014-08-08T10:44:44.993 INFO:tasks.rados.rados.0.vpm167.stdout:3595: done (1 left) 2014-08-08T10:44:44.993 INFO:tasks.rados.rados.0.vpm167.stdout:3596: done (0 left) 2014-08-08T10:44:44.993 INFO:tasks.rados.rados.0.vpm167.stdout:3597: snap_remove snap 440 2014-08-08T10:44:46.339 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 3, 2, 5] out_osds: [0] dead_osds: [] live_osds: [1, 0, 5, 4, 2, 3] 2014-08-08T10:44:46.339 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T10:44:47.138 INFO:tasks.rados.rados.0.vpm167.stdout:3597: done (0 left) 2014-08-08T10:44:47.139 INFO:tasks.rados.rados.0.vpm167.stdout:3598: rollback oid 42 current snap is 474 2014-08-08T10:44:47.139 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 42 to 473 2014-08-08T10:44:47.139 INFO:tasks.rados.rados.0.vpm167.stdout:3599: rollback oid 5 current snap is 474 2014-08-08T10:44:47.140 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 5 to 442 2014-08-08T10:44:47.140 INFO:tasks.rados.rados.0.vpm167.stdout:3600: delete oid 47 current snap is 474 2014-08-08T10:44:47.142 INFO:tasks.rados.rados.0.vpm167.stdout:3600: done (2 left) 2014-08-08T10:44:47.142 INFO:tasks.rados.rados.0.vpm167.stdout:3601: write oid 31 current snap is 474 2014-08-08T10:44:47.143 INFO:tasks.rados.rados.0.vpm167.stdout:3601: seq_num 1354 ranges {476300=699656,1589250=716521,2476299=1} 2014-08-08T10:44:47.179 INFO:tasks.rados.rados.0.vpm167.stdout:3601: writing vpm1675861-31 from 476300 to 1175956 tid 1 2014-08-08T10:44:47.239 INFO:tasks.rados.rados.0.vpm167.stdout:3601: writing vpm1675861-31 from 1589250 to 2305771 tid 2 2014-08-08T10:44:47.244 INFO:tasks.rados.rados.0.vpm167.stdout:3601: writing vpm1675861-31 from 2476299 to 2476300 tid 3 2014-08-08T10:44:47.246 INFO:tasks.rados.rados.0.vpm167.stdout:3602: read oid 45 snap 422 2014-08-08T10:44:47.247 INFO:tasks.rados.rados.0.vpm167.stdout:3603: rollback oid 32 current snap is 474 2014-08-08T10:44:47.247 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 32 to 422 2014-08-08T10:44:47.247 INFO:tasks.rados.rados.0.vpm167.stdout:3604: delete oid 20 current snap is 474 2014-08-08T10:44:47.249 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 32 v 0 (ObjNum -1043019784 snap 32533 seq_num 0) dirty dne 2014-08-08T10:44:47.269 INFO:tasks.rados.rados.0.vpm167.stdout:3603: done (5 left) 2014-08-08T10:44:47.269 INFO:tasks.rados.rados.0.vpm167.stdout:3604: done (4 left) 2014-08-08T10:44:47.270 INFO:tasks.rados.rados.0.vpm167.stdout:3605: read oid 14 snap -1 2014-08-08T10:44:47.270 INFO:tasks.rados.rados.0.vpm167.stdout:3606: snap_create 2014-08-08T10:44:47.279 INFO:tasks.rados.rados.0.vpm167.stdout:3605: expect (ObjNum 1334 snap 469 seq_num 1334) 2014-08-08T10:44:48.019 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 5 v 1334 (ObjNum 1005 snap 342 seq_num 1005) dirty exists 2014-08-08T10:44:48.020 INFO:tasks.rados.rados.0.vpm167.stdout:3601: finishing write tid 1 to vpm1675861-31 2014-08-08T10:44:48.021 INFO:tasks.rados.rados.0.vpm167.stdout:3601: finishing write tid 2 to vpm1675861-31 2014-08-08T10:44:48.021 INFO:tasks.rados.rados.0.vpm167.stdout:3601: finishing write tid 3 to vpm1675861-31 2014-08-08T10:44:48.022 INFO:tasks.rados.rados.0.vpm167.stdout:3601: finishing write tid 5 to vpm1675861-31 2014-08-08T10:44:49.054 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T10:44:49.074 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-08T10:44:49.438 INFO:tasks.rados.rados.0.vpm167.stdout:3599: done (5 left) 2014-08-08T10:44:49.438 INFO:tasks.rados.rados.0.vpm167.stdout:3602: done (4 left) 2014-08-08T10:44:49.439 INFO:tasks.rados.rados.0.vpm167.stdout:3605: done (3 left) 2014-08-08T10:44:49.439 INFO:tasks.rados.rados.0.vpm167.stdout:3606: done (2 left) 2014-08-08T10:44:49.439 INFO:tasks.rados.rados.0.vpm167.stdout:3607: delete oid 18 current snap is 475 2014-08-08T10:44:50.806 INFO:tasks.rados.rados.0.vpm167.stdout:3607: done (2 left) 2014-08-08T10:44:50.806 INFO:tasks.rados.rados.0.vpm167.stdout:3608: delete oid 38 current snap is 475 2014-08-08T10:44:50.807 INFO:tasks.rados.rados.0.vpm167.stdout:3608: done (2 left) 2014-08-08T10:44:50.810 INFO:tasks.rados.rados.0.vpm167.stdout:3609: rollback oid 43 current snap is 475 2014-08-08T10:44:50.810 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 43 to 460 2014-08-08T10:44:50.810 INFO:tasks.rados.rados.0.vpm167.stdout:3610: write oid 33 current snap is 475 2014-08-08T10:44:50.810 INFO:tasks.rados.rados.0.vpm167.stdout:3610: seq_num 1358 ranges {522371=400000} 2014-08-08T10:44:50.824 INFO:tasks.rados.rados.0.vpm167.stdout:3610: writing vpm1675861-33 from 522371 to 922371 tid 1 2014-08-08T10:44:50.831 INFO:tasks.rados.rados.0.vpm167.stdout:3611: snap_create 2014-08-08T10:44:51.219 INFO:tasks.rados.rados.0.vpm167.stdout:3610: finishing write tid 1 to vpm1675861-33 2014-08-08T10:44:51.219 INFO:tasks.rados.rados.0.vpm167.stdout:3610: finishing write tid 3 to vpm1675861-33 2014-08-08T10:44:51.219 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 43 v 897 (ObjNum 1 snap 0 seq_num 32) dirty dne 2014-08-08T10:44:51.242 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T10:44:51.278 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T10:44:51.494 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 2243 (ObjNum 1336 snap 469 seq_num 1336) dirty exists 2014-08-08T10:44:51.894 INFO:tasks.rados.rados.0.vpm167.stdout:3598: done (4 left) 2014-08-08T10:44:51.894 INFO:tasks.rados.rados.0.vpm167.stdout:3609: done (3 left) 2014-08-08T10:44:51.895 INFO:tasks.rados.rados.0.vpm167.stdout:3611: done (2 left) 2014-08-08T10:44:51.895 INFO:tasks.rados.rados.0.vpm167.stdout:3612: rollback oid 40 current snap is 476 2014-08-08T10:44:51.895 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 40 to 452 2014-08-08T10:44:51.895 INFO:tasks.rados.rados.0.vpm167.stdout:3613: read oid 43 snap -1 2014-08-08T10:44:51.896 INFO:tasks.rados.rados.0.vpm167.stdout:3614: write oid 13 current snap is 476 2014-08-08T10:44:51.896 INFO:tasks.rados.rados.0.vpm167.stdout:3614: seq_num 1359 ranges {617599=1} 2014-08-08T10:44:51.903 INFO:tasks.rados.rados.0.vpm167.stdout:3614: writing vpm1675861-13 from 617599 to 617600 tid 1 2014-08-08T10:44:51.904 INFO:tasks.rados.rados.0.vpm167.stdout:3615: write oid 42 current snap is 476 2014-08-08T10:44:51.904 INFO:tasks.rados.rados.0.vpm167.stdout:3615: seq_num 1360 ranges {562430=719026,1362429=1} 2014-08-08T10:44:51.937 INFO:tasks.rados.rados.0.vpm167.stdout:3615: writing vpm1675861-42 from 562430 to 1281456 tid 1 2014-08-08T10:44:51.949 INFO:tasks.rados.rados.0.vpm167.stdout:3615: writing vpm1675861-42 from 1362429 to 1362430 tid 2 2014-08-08T10:44:51.950 INFO:tasks.rados.rados.0.vpm167.stdout:3616: rollback oid 14 current snap is 476 2014-08-08T10:44:51.950 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 14 to 459 2014-08-08T10:44:51.950 INFO:tasks.rados.rados.0.vpm167.stdout:3617: delete oid 26 current snap is 476 2014-08-08T10:44:52.233 INFO:tasks.rados.rados.0.vpm167.stdout:3614: finishing write tid 1 to vpm1675861-13 2014-08-08T10:44:52.375 INFO:tasks.rados.rados.0.vpm167.stdout:3615: finishing write tid 1 to vpm1675861-42 2014-08-08T10:44:52.376 INFO:tasks.rados.rados.0.vpm167.stdout:3614: finishing write tid 3 to vpm1675861-13 2014-08-08T10:44:53.125 INFO:tasks.rados.rados.0.vpm167.stdout:3615: finishing write tid 2 to vpm1675861-42 2014-08-08T10:44:54.047 INFO:tasks.rados.rados.0.vpm167.stdout:3610: finishing write tid 4 to vpm1675861-33 2014-08-08T10:44:54.047 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 1979 (ObjNum 1358 snap 475 seq_num 1358) dirty exists 2014-08-08T10:44:54.103 INFO:tasks.rados.rados.0.vpm167.stdout:3615: finishing write tid 4 to vpm1675861-42 2014-08-08T10:44:54.104 INFO:tasks.rados.rados.0.vpm167.stdout:3610: done (7 left) 2014-08-08T10:44:54.104 INFO:tasks.rados.rados.0.vpm167.stdout:3613: done (6 left) 2014-08-08T10:44:54.104 INFO:tasks.rados.rados.0.vpm167.stdout:3617: done (5 left) 2014-08-08T10:44:54.105 INFO:tasks.rados.rados.0.vpm167.stdout:3618: write oid 30 current snap is 476 2014-08-08T10:44:54.105 INFO:tasks.rados.rados.0.vpm167.stdout:3618: seq_num 1362 ranges {718842=697707,2057123=261719} 2014-08-08T10:44:54.110 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T10:44:54.122 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T10:44:54.142 INFO:tasks.rados.rados.0.vpm167.stdout:3618: writing vpm1675861-30 from 718842 to 1416549 tid 1 2014-08-08T10:44:54.167 INFO:tasks.rados.rados.0.vpm167.stdout:3618: writing vpm1675861-30 from 2057123 to 2318842 tid 2 2014-08-08T10:44:54.168 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 40 v 943 (ObjNum 1292 snap 449 seq_num 1292) dirty exists 2014-08-08T10:44:54.168 INFO:tasks.rados.rados.0.vpm167.stdout:3612: done (5 left) 2014-08-08T10:44:54.169 INFO:tasks.rados.rados.0.vpm167.stdout:3619: read oid 38 snap -1 2014-08-08T10:44:54.169 INFO:tasks.rados.rados.0.vpm167.stdout:3620: snap_remove snap 465 2014-08-08T10:44:54.364 INFO:tasks.rados.rados.0.vpm167.stdout:3601: finishing write tid 6 to vpm1675861-31 2014-08-08T10:44:54.365 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 31 v 1424 (ObjNum 1354 snap 474 seq_num 1354) dirty exists 2014-08-08T10:44:54.372 INFO:tasks.rados.rados.0.vpm167.stdout:3618: finishing write tid 1 to vpm1675861-30 2014-08-08T10:44:54.869 INFO:tasks.rados.rados.0.vpm167.stdout:3618: finishing write tid 2 to vpm1675861-30 2014-08-08T10:44:54.870 INFO:tasks.rados.rados.0.vpm167.stdout:3618: finishing write tid 4 to vpm1675861-30 2014-08-08T10:44:55.641 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 14 v 2249 (ObjNum 1172 snap 412 seq_num 1172) dirty exists 2014-08-08T10:44:56.102 INFO:tasks.rados.rados.0.vpm167.stdout:3614: finishing write tid 4 to vpm1675861-13 2014-08-08T10:44:56.102 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 13 v 1982 (ObjNum 1359 snap 476 seq_num 1359) dirty exists 2014-08-08T10:44:56.154 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T10:44:56.195 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T10:44:56.520 INFO:tasks.rados.rados.0.vpm167.stdout:3601: done (6 left) 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3614: done (5 left) 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3616: done (4 left) 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3619: done (3 left) 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3620: done (2 left) 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3621: write oid 3 current snap is 476 2014-08-08T10:44:56.526 INFO:tasks.rados.rados.0.vpm167.stdout:3621: seq_num 1363 ranges {662647=421959,1771254=486198,2980863=81784} 2014-08-08T10:44:56.543 INFO:tasks.rados.rados.0.vpm167.stdout:3621: writing vpm1675861-3 from 662647 to 1084606 tid 1 2014-08-08T10:44:56.568 INFO:tasks.rados.rados.0.vpm167.stdout:3621: writing vpm1675861-3 from 1771254 to 2257452 tid 2 2014-08-08T10:44:56.577 INFO:tasks.rados.rados.0.vpm167.stdout:3621: writing vpm1675861-3 from 2980863 to 3062647 tid 3 2014-08-08T10:44:56.578 INFO:tasks.rados.rados.0.vpm167.stdout:3622: read oid 29 snap -1 2014-08-08T10:44:56.579 INFO:tasks.rados.rados.0.vpm167.stdout:3623: read oid 12 snap -1 2014-08-08T10:44:56.579 INFO:tasks.rados.rados.0.vpm167.stdout:3624: snap_remove snap 446 2014-08-08T10:44:56.621 INFO:tasks.rados.rados.0.vpm167.stdout:3623: expect (ObjNum 1301 snap 456 seq_num 1301) 2014-08-08T10:44:57.710 INFO:teuthology.orchestra.run.vpm166.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T10:44:57.745 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T10:44:57.745 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:44:58.456 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:44:58.525 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:44:58.768 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:44:58.830 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:44:59.205 INFO:tasks.rados.rados.0.vpm167.stdout:3622: done (5 left) 2014-08-08T10:44:59.206 INFO:tasks.rados.rados.0.vpm167.stdout:3623: done (4 left) 2014-08-08T10:44:59.206 INFO:tasks.rados.rados.0.vpm167.stdout:3624: done (3 left) 2014-08-08T10:44:59.207 INFO:tasks.rados.rados.0.vpm167.stdout:3625: write oid 33 current snap is 476 2014-08-08T10:44:59.207 INFO:tasks.rados.rados.0.vpm167.stdout:3625: seq_num 1364 ranges {606453=793763,2132938=556393,3181064=625389} 2014-08-08T10:44:59.233 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:44:19.623634', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88318736, u'health': u'HEALTH_OK', u'kb_used': 10634832, u'store_stats': {u'bytes_total': 30663153, u'bytes_log': 4552959, u'last_updated': u'0.000000', u'bytes_misc': 26110194, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:44:26.568730', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87167228, u'health': u'HEALTH_OK', u'kb_used': 11786340, u'store_stats': {u'bytes_total': 33125843, u'bytes_log': 7029724, u'last_updated': u'0.000000', u'bytes_misc': 26096119, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:44:23.353375', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88317696, u'health': u'HEALTH_OK', u'kb_used': 10635872, u'store_stats': {u'bytes_total': 32960229, u'bytes_log': 6843944, u'last_updated': u'0.000000', u'bytes_misc': 26116285, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 81/834 objects degraded (9.712%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 54'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 5455331, u'degraded_objects': 81, u'num_pgs': 358, u'recovering_keys_per_sec': 0, u'data_bytes': 368980063, u'degraded_total': 834, u'bytes_used': 1765797888, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 902628, u'op_per_sec': 8, u'version': 2534, u'pgs_by_state': [{u'count': 354, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'read_bytes_sec': 1648, u'degraded_ratio': u'9.712', u'bytes_avail': 1071451738112}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1372, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:44:59.234 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:44:59.253 INFO:tasks.rados.rados.0.vpm167.stdout:3625: writing vpm1675861-33 from 606453 to 1400216 tid 1 2014-08-08T10:44:59.297 INFO:tasks.rados.rados.0.vpm167.stdout:3625: writing vpm1675861-33 from 2132938 to 2689331 tid 2 2014-08-08T10:44:59.336 INFO:tasks.rados.rados.0.vpm167.stdout:3625: writing vpm1675861-33 from 3181064 to 3806453 tid 3 2014-08-08T10:44:59.339 INFO:tasks.rados.rados.0.vpm167.stdout:3621: finishing write tid 1 to vpm1675861-3 2014-08-08T10:44:59.339 INFO:tasks.rados.rados.0.vpm167.stdout:3621: finishing write tid 2 to vpm1675861-3 2014-08-08T10:44:59.340 INFO:tasks.rados.rados.0.vpm167.stdout:3621: finishing write tid 3 to vpm1675861-3 2014-08-08T10:44:59.340 INFO:tasks.rados.rados.0.vpm167.stdout:3621: finishing write tid 5 to vpm1675861-3 2014-08-08T10:44:59.340 INFO:tasks.rados.rados.0.vpm167.stdout:3615: finishing write tid 5 to vpm1675861-42 2014-08-08T10:44:59.340 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 42 v 2247 (ObjNum 1360 snap 476 seq_num 1360) dirty exists 2014-08-08T10:44:59.341 INFO:tasks.rados.rados.0.vpm167.stdout:3618: finishing write tid 5 to vpm1675861-30 2014-08-08T10:44:59.341 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 30 v 1338 (ObjNum 1362 snap 476 seq_num 1362) dirty exists 2014-08-08T10:44:59.341 INFO:tasks.rados.rados.0.vpm167.stdout:3621: finishing write tid 6 to vpm1675861-3 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 3 v 920 (ObjNum 1363 snap 476 seq_num 1363) dirty exists 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:3625: finishing write tid 1 to vpm1675861-33 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:3615: done (3 left) 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:3618: done (2 left) 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:3621: done (1 left) 2014-08-08T10:44:59.342 INFO:tasks.rados.rados.0.vpm167.stdout:3626: write oid 41 current snap is 476 2014-08-08T10:44:59.346 INFO:tasks.rados.rados.0.vpm167.stdout:3626: seq_num 1365 ranges {702707=400640,1847071=778012,3382290=120417} 2014-08-08T10:44:59.369 INFO:tasks.rados.rados.0.vpm167.stdout:3626: writing vpm1675861-41 from 702707 to 1103347 tid 1 2014-08-08T10:44:59.423 INFO:tasks.rados.rados.0.vpm167.stdout:3626: writing vpm1675861-41 from 1847071 to 2625083 tid 2 2014-08-08T10:44:59.440 INFO:tasks.rados.rados.0.vpm167.stdout:3626: writing vpm1675861-41 from 3382290 to 3502707 tid 3 2014-08-08T10:44:59.442 INFO:tasks.rados.rados.0.vpm167.stdout:3625: finishing write tid 2 to vpm1675861-33 2014-08-08T10:44:59.442 INFO:tasks.rados.rados.0.vpm167.stdout:3625: finishing write tid 3 to vpm1675861-33 2014-08-08T10:44:59.442 INFO:tasks.rados.rados.0.vpm167.stdout:3625: finishing write tid 5 to vpm1675861-33 2014-08-08T10:44:59.442 INFO:tasks.rados.rados.0.vpm167.stdout:3626: finishing write tid 1 to vpm1675861-41 2014-08-08T10:44:59.442 INFO:tasks.rados.rados.0.vpm167.stdout:3625: finishing write tid 6 to vpm1675861-33 2014-08-08T10:44:59.443 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 33 v 2001 (ObjNum 1364 snap 476 seq_num 1364) dirty exists 2014-08-08T10:44:59.443 INFO:tasks.rados.rados.0.vpm167.stdout:3625: done (1 left) 2014-08-08T10:44:59.443 INFO:tasks.rados.rados.0.vpm167.stdout:3627: snap_create 2014-08-08T10:44:59.511 INFO:tasks.rados.rados.0.vpm167.stdout:3626: finishing write tid 2 to vpm1675861-41 2014-08-08T10:44:59.535 INFO:tasks.rados.rados.0.vpm167.stdout:3626: finishing write tid 3 to vpm1675861-41 2014-08-08T10:44:59.552 INFO:tasks.rados.rados.0.vpm167.stdout:3626: finishing write tid 5 to vpm1675861-41 2014-08-08T10:44:59.561 INFO:tasks.rados.rados.0.vpm167.stdout:3626: finishing write tid 6 to vpm1675861-41 2014-08-08T10:44:59.561 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1906 (ObjNum 1365 snap 476 seq_num 1365) dirty exists 2014-08-08T10:44:59.586 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:44:59.586 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:44:59.813 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:45:00.470 INFO:tasks.rados.rados.0.vpm167.stdout:3626: done (1 left) 2014-08-08T10:45:00.470 INFO:tasks.rados.rados.0.vpm167.stdout:3627: done (0 left) 2014-08-08T10:45:00.471 INFO:tasks.rados.rados.0.vpm167.stdout:3628: delete oid 37 current snap is 477 2014-08-08T10:45:01.056 INFO:tasks.rados.rados.0.vpm167.stdout:3628: done (0 left) 2014-08-08T10:45:01.057 INFO:tasks.rados.rados.0.vpm167.stdout:3629: read oid 31 snap -1 2014-08-08T10:45:01.057 INFO:tasks.rados.rados.0.vpm167.stdout:3630: delete oid 17 current snap is 477 2014-08-08T10:45:01.092 INFO:tasks.rados.rados.0.vpm167.stdout:3630: done (1 left) 2014-08-08T10:45:01.093 INFO:tasks.rados.rados.0.vpm167.stdout:3631: rollback oid 47 current snap is 477 2014-08-08T10:45:01.093 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 47 to 448 2014-08-08T10:45:01.093 INFO:tasks.rados.rados.0.vpm167.stdout:3632: read oid 37 snap -1 2014-08-08T10:45:01.097 INFO:tasks.rados.rados.0.vpm167.stdout:3633: write oid 41 current snap is 477 2014-08-08T10:45:01.097 INFO:tasks.rados.rados.0.vpm167.stdout:3633: seq_num 1368 ranges {403948=718101,1203947=1} 2014-08-08T10:45:01.143 INFO:tasks.rados.rados.0.vpm167.stdout:3633: writing vpm1675861-41 from 403948 to 1122049 tid 1 2014-08-08T10:45:01.147 INFO:tasks.rados.rados.0.vpm167.stdout:3633: writing vpm1675861-41 from 1203947 to 1203948 tid 2 2014-08-08T10:45:01.148 INFO:tasks.rados.rados.0.vpm167.stdout:3629: expect (ObjNum 1354 snap 474 seq_num 1354) 2014-08-08T10:45:02.552 INFO:tasks.rados.rados.0.vpm167.stdout:3629: done (3 left) 2014-08-08T10:45:02.552 INFO:tasks.rados.rados.0.vpm167.stdout:3634: read oid 38 snap -1 2014-08-08T10:45:02.552 INFO:tasks.rados.rados.0.vpm167.stdout:3635: write oid 17 current snap is 477 2014-08-08T10:45:02.553 INFO:tasks.rados.rados.0.vpm167.stdout:3635: seq_num 1369 ranges {747754=724977,1947753=1} 2014-08-08T10:45:02.580 INFO:tasks.rados.rados.0.vpm167.stdout:3635: writing vpm1675861-17 from 747754 to 1472731 tid 1 2014-08-08T10:45:02.589 INFO:tasks.rados.rados.0.vpm167.stdout:3635: writing vpm1675861-17 from 1947753 to 1947754 tid 2 2014-08-08T10:45:02.591 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 47 v 1474 (ObjNum 1255 snap 438 seq_num 1255) dirty exists 2014-08-08T10:45:02.591 INFO:tasks.rados.rados.0.vpm167.stdout:3633: finishing write tid 1 to vpm1675861-41 2014-08-08T10:45:02.591 INFO:tasks.rados.rados.0.vpm167.stdout:3633: finishing write tid 2 to vpm1675861-41 2014-08-08T10:45:02.591 INFO:tasks.rados.rados.0.vpm167.stdout:3633: finishing write tid 4 to vpm1675861-41 2014-08-08T10:45:02.591 INFO:tasks.rados.rados.0.vpm167.stdout:3633: finishing write tid 5 to vpm1675861-41 2014-08-08T10:45:02.592 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 41 v 1910 (ObjNum 1368 snap 477 seq_num 1368) dirty exists 2014-08-08T10:45:02.592 INFO:tasks.rados.rados.0.vpm167.stdout:3631: done (4 left) 2014-08-08T10:45:02.592 INFO:tasks.rados.rados.0.vpm167.stdout:3633: done (3 left) 2014-08-08T10:45:02.592 INFO:tasks.rados.rados.0.vpm167.stdout:3634: done (2 left) 2014-08-08T10:45:02.593 INFO:tasks.rados.rados.0.vpm167.stdout:3636: write oid 30 current snap is 477 2014-08-08T10:45:02.593 INFO:tasks.rados.rados.0.vpm167.stdout:3636: seq_num 1370 ranges {444008=697805,1644007=1} 2014-08-08T10:45:02.616 INFO:tasks.rados.rados.0.vpm167.stdout:3636: writing vpm1675861-30 from 444008 to 1141813 tid 1 2014-08-08T10:45:02.627 INFO:tasks.rados.rados.0.vpm167.stdout:3636: writing vpm1675861-30 from 1644007 to 1644008 tid 2 2014-08-08T10:45:02.628 INFO:tasks.rados.rados.0.vpm167.stdout:3637: write oid 23 current snap is 477 2014-08-08T10:45:02.628 INFO:tasks.rados.rados.0.vpm167.stdout:3637: seq_num 1371 ranges {504165=421034,1625624=585679,2688842=215323} 2014-08-08T10:45:02.645 INFO:tasks.rados.rados.0.vpm167.stdout:3637: writing vpm1675861-23 from 504165 to 925199 tid 1 2014-08-08T10:45:02.670 INFO:tasks.rados.rados.0.vpm167.stdout:3637: writing vpm1675861-23 from 1625624 to 2211303 tid 2 2014-08-08T10:45:02.682 INFO:tasks.rados.rados.0.vpm167.stdout:3637: writing vpm1675861-23 from 2688842 to 2904165 tid 3 2014-08-08T10:45:02.683 INFO:tasks.rados.rados.0.vpm167.stdout:3638: read oid 13 snap 471 2014-08-08T10:45:02.683 INFO:tasks.rados.rados.0.vpm167.stdout:3639: snap_remove snap 466 2014-08-08T10:45:02.739 INFO:tasks.rados.rados.0.vpm167.stdout:3635: finishing write tid 1 to vpm1675861-17 2014-08-08T10:45:02.860 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:45:02.996 INFO:tasks.rados.rados.0.vpm167.stdout:3635: finishing write tid 2 to vpm1675861-17 2014-08-08T10:45:02.997 INFO:tasks.rados.rados.0.vpm167.stdout:3635: finishing write tid 4 to vpm1675861-17 2014-08-08T10:45:03.001 INFO:tasks.rados.rados.0.vpm167.stdout:3636: finishing write tid 1 to vpm1675861-30 2014-08-08T10:45:03.003 INFO:tasks.rados.rados.0.vpm167.stdout:3636: finishing write tid 2 to vpm1675861-30 2014-08-08T10:45:03.003 INFO:tasks.rados.rados.0.vpm167.stdout:3636: finishing write tid 4 to vpm1675861-30 2014-08-08T10:45:03.005 INFO:tasks.rados.rados.0.vpm167.stdout:3637: finishing write tid 1 to vpm1675861-23 2014-08-08T10:45:03.006 INFO:tasks.rados.rados.0.vpm167.stdout:3637: finishing write tid 2 to vpm1675861-23 2014-08-08T10:45:03.007 INFO:tasks.rados.rados.0.vpm167.stdout:3637: finishing write tid 3 to vpm1675861-23 2014-08-08T10:45:03.009 INFO:tasks.rados.rados.0.vpm167.stdout:3637: finishing write tid 5 to vpm1675861-23 2014-08-08T10:45:03.102 INFO:tasks.rados.rados.0.vpm167.stdout:3635: finishing write tid 5 to vpm1675861-17 2014-08-08T10:45:03.102 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 17 v 2378 (ObjNum 1369 snap 477 seq_num 1369) dirty exists 2014-08-08T10:45:03.397 INFO:tasks.rados.rados.0.vpm167.stdout:3635: done (5 left) 2014-08-08T10:45:03.397 INFO:tasks.rados.rados.0.vpm167.stdout:3638: done (4 left) 2014-08-08T10:45:03.397 INFO:tasks.rados.rados.0.vpm167.stdout:3639: done (3 left) 2014-08-08T10:45:03.397 INFO:tasks.rados.rados.0.vpm167.stdout:3640: read oid 19 snap -1 2014-08-08T10:45:03.398 INFO:tasks.rados.rados.0.vpm167.stdout:3641: read oid 28 snap 458 2014-08-08T10:45:03.398 INFO:tasks.rados.rados.0.vpm167.stdout:3642: delete oid 39 current snap is 477 2014-08-08T10:45:03.403 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:45:03.542 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:45:03.543 INFO:tasks.rados.rados.0.vpm167.stdout:3641: expect (ObjNum 1252 snap 438 seq_num 1252) 2014-08-08T10:45:03.859 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 12, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.240:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.180:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.240:6790/0'}], u'modified': u'2014-08-08 16:47:35.225343', u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7', u'created': u'2014-08-08 16:47:35.225343'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 12, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004798', u'skew': u'-0.000010', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.007954', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:44:19.623634', u'name': u'a', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88318736, u'health': u'HEALTH_OK', u'kb_used': 10634832, u'store_stats': {u'bytes_total': 30663153, u'bytes_log': 4552959, u'last_updated': u'0.000000', u'bytes_misc': 26110194, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:44:26.568730', u'name': u'c', u'avail_percent': 84, u'kb_total': 103178528, u'kb_avail': 87167228, u'health': u'HEALTH_OK', u'kb_used': 11786340, u'store_stats': {u'bytes_total': 33125843, u'bytes_log': 7029724, u'last_updated': u'0.000000', u'bytes_misc': 26096119, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:44:23.353375', u'name': u'b', u'avail_percent': 85, u'kb_total': 103178528, u'kb_avail': 88317696, u'health': u'HEALTH_OK', u'kb_used': 10635872, u'store_stats': {u'bytes_total': 32960229, u'bytes_log': 6843944, u'last_updated': u'0.000000', u'bytes_misc': 26116285, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 28/844 objects degraded (3.318%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 54'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_1 pg_num 56 > pgp_num 36'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 16661465, u'degraded_objects': 28, u'num_pgs': 358, u'recovering_keys_per_sec': 0, u'data_bytes': 380849833, u'degraded_total': 844, u'bytes_used': 1814945792, u'recovering_objects_per_sec': 7, u'write_bytes_sec': 1595756, u'op_per_sec': 12, u'version': 2538, u'pgs_by_state': [{u'count': 355, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'read_bytes_sec': 1982904, u'degraded_ratio': u'3.318', u'bytes_avail': 1071402590208}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 1374, u'num_in_osds': 5}}, u'fsid': u'8ab7f27c-cc32-4e68-9458-c5119c980af7'} 2014-08-08T10:45:03.860 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:45:04.042 INFO:tasks.rados.rados.0.vpm167.stdout:3640: done (5 left) 2014-08-08T10:45:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3641: done (4 left) 2014-08-08T10:45:04.043 INFO:tasks.rados.rados.0.vpm167.stdout:3642: done (3 left) 2014-08-08T10:45:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3643: write oid 21 current snap is 477 2014-08-08T10:45:04.044 INFO:tasks.rados.rados.0.vpm167.stdout:3643: seq_num 1373 ranges {544225=400738,1702464=476470,2973917=370308} 2014-08-08T10:45:04.059 INFO:tasks.rados.rados.0.vpm167.stdout:3643: writing vpm1675861-21 from 544225 to 944963 tid 1 2014-08-08T10:45:04.081 INFO:tasks.rados.rados.0.vpm167.stdout:3643: writing vpm1675861-21 from 1702464 to 2178934 tid 2 2014-08-08T10:45:04.100 INFO:tasks.rados.rados.0.vpm167.stdout:3643: writing vpm1675861-21 from 2973917 to 3344225 tid 3 2014-08-08T10:45:04.101 INFO:tasks.rados.rados.0.vpm167.stdout:3644: write oid 4 current snap is 477 2014-08-08T10:45:04.101 INFO:tasks.rados.rados.0.vpm167.stdout:3644: seq_num 1374 ranges {88029=1} 2014-08-08T10:45:04.102 INFO:tasks.rados.rados.0.vpm167.stdout:3644: writing vpm1675861-4 from 88029 to 88030 tid 1 2014-08-08T10:45:04.103 INFO:tasks.rados.rados.0.vpm167.stdout:3645: rollback oid 48 current snap is 477 2014-08-08T10:45:04.103 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 48 to 445 2014-08-08T10:45:04.104 INFO:tasks.rados.rados.0.vpm167.stdout:3646: read oid 36 snap 344 2014-08-08T10:45:04.104 INFO:tasks.rados.rados.0.vpm167.stdout:3647: read oid 15 snap -1 2014-08-08T10:45:04.105 INFO:tasks.rados.rados.0.vpm167.stdout:3648: delete oid 40 current snap is 477 2014-08-08T10:45:04.126 INFO:tasks.rados.rados.0.vpm167.stdout:3647: expect (ObjNum 1341 snap 469 seq_num 1341) 2014-08-08T10:45:04.183 INFO:tasks.rados.rados.0.vpm167.stdout:3646: expect (ObjNum 956 snap 316 seq_num 956) 2014-08-08T10:45:04.186 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:45:04.187 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:45:04.370 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 48 v 2007 (ObjNum 1269 snap 440 seq_num 1269) dirty exists 2014-08-08T10:45:04.371 INFO:tasks.rados.rados.0.vpm167.stdout:3643: finishing write tid 1 to vpm1675861-21 2014-08-08T10:45:04.424 INFO:teuthology.orchestra.run.vpm166.stderr:dumped all in format json 2014-08-08T10:45:04.799 INFO:tasks.rados.rados.0.vpm167.stdout:3643: finishing write tid 2 to vpm1675861-21 2014-08-08T10:45:04.799 INFO:tasks.rados.rados.0.vpm167.stdout:3643: finishing write tid 3 to vpm1675861-21 2014-08-08T10:45:04.809 INFO:tasks.rados.rados.0.vpm167.stdout:3643: finishing write tid 5 to vpm1675861-21 2014-08-08T10:45:04.824 INFO:tasks.rados.rados.0.vpm167.stdout:3645: done (8 left) 2014-08-08T10:45:04.824 INFO:tasks.rados.rados.0.vpm167.stdout:3646: done (7 left) 2014-08-08T10:45:04.824 INFO:tasks.rados.rados.0.vpm167.stdout:3647: done (6 left) 2014-08-08T10:45:04.824 INFO:tasks.rados.rados.0.vpm167.stdout:3648: done (5 left) 2014-08-08T10:45:04.824 INFO:tasks.rados.rados.0.vpm167.stdout:3649: rollback oid 3 current snap is 477 2014-08-08T10:45:04.825 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 3 to 384 2014-08-08T10:45:04.825 INFO:tasks.rados.rados.0.vpm167.stdout:3650: write oid 34 current snap is 477 2014-08-08T10:45:04.825 INFO:tasks.rados.rados.0.vpm167.stdout:3650: seq_num 1376 ranges {645466=400000} 2014-08-08T10:45:04.846 INFO:tasks.rados.rados.0.vpm167.stdout:3650: writing vpm1675861-34 from 645466 to 1045466 tid 1 2014-08-08T10:45:04.848 INFO:tasks.rados.rados.0.vpm167.stdout:3644: finishing write tid 1 to vpm1675861-4 2014-08-08T10:45:04.848 INFO:tasks.rados.rados.0.vpm167.stdout:3644: finishing write tid 3 to vpm1675861-4 2014-08-08T10:45:04.848 INFO:tasks.rados.rados.0.vpm167.stdout:3651: rollback oid 29 current snap is 477 2014-08-08T10:45:04.848 INFO:tasks.rados.rados.0.vpm167.stdout:rollback oid 29 to 472 2014-08-08T10:45:04.849 INFO:tasks.rados.rados.0.vpm167.stdout:3652: read oid 41 snap -1 2014-08-08T10:45:04.849 INFO:tasks.rados.rados.0.vpm167.stdout:3653: snap_create 2014-08-08T10:45:04.862 INFO:tasks.rados.rados.0.vpm167.stdout:update_object_version oid 29 v 0 (ObjNum 31574888 snap 0 seq_num 39699552) dirty dne 2014-08-08T10:45:04.881 INFO:tasks.rados.rados.0.vpm167.stdout:3652: expect (ObjNum 1368 snap 477 seq_num 1368) 2014-08-08T10:45:05.299 INFO:teuthology.orchestra.run.vpm167.stderr:testContainerListing (test.functional.tests.TestContainerPaths) ... ERROR ---------------------------------------------------------------------- 2014-08-08T12:34:01.663 INFO:teuthology.orchestra.run.vpm134.stderr:testIfUnmodifiedSince (test.functional.tests.TestFileComparisonUTF8) ... ok 2014-08-08T12:34:01.796 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T12:34:01.797 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:34:02.973 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:34:06.501 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:34:07.380 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:34:07.649 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:34:07.876 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 42, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.043298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.006208', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:33:17.806977', u'name': u'a', u'avail_percent': 71, u'kb_total': 103178528, u'kb_avail': 73617856, u'health': u'HEALTH_OK', u'kb_used': 25335712, u'store_stats': {u'bytes_total': 62676111, u'bytes_log': 11299431, u'last_updated': u'0.000000', u'bytes_misc': 51376680, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:33:52.471405', u'name': u'c', u'avail_percent': 74, u'kb_total': 103178528, u'kb_avail': 76588976, u'health': u'HEALTH_OK', u'kb_used': 22364592, u'store_stats': {u'bytes_total': 51816680, u'bytes_log': 2530646, u'last_updated': u'0.000000', u'bytes_misc': 49286034, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:33:24.971230', u'name': u'b', u'avail_percent': 71, u'kb_total': 103178528, u'kb_avail': 73603064, u'health': u'HEALTH_OK', u'kb_used': 25350504, u'store_stats': {u'bytes_total': 69281523, u'bytes_log': 17909207, u'last_updated': u'0.000000', u'bytes_misc': 51372316, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 284 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 284 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1116, u'data_bytes': 1009771, u'bytes_used': 1203793920, u'op_per_sec': 96, u'version': 6139, u'pgs_by_state': [{u'count': 1115, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'read_bytes_sec': 45905, u'write_bytes_sec': 19127, u'bytes_avail': 1286657249280}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2883, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:34:07.876 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T12:34:07.877 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:07.877 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [1, 5, 0, 2, 4, 3] 2014-08-08T12:34:07.877 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3' 2014-08-08T12:34:09.943 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.3. 2014-08-08T12:34:15.030 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 0, 2, 4] out_osds: [3] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:15.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:15.030 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 5, 0, 2, 4] 2014-08-08T12:34:15.031 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-08T12:34:17.293 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.0. 2014-08-08T12:34:22.432 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4] out_osds: [3, 0] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:22.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:22.432 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T12:34:22.432 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T12:34:22.432 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T12:34:22.433 INFO:teuthology.orchestra.run.vpm132:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T12:34:27.943 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4] out_osds: [3, 0] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:27.943 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:27.943 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-08T12:34:27.943 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T12:34:27.944 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T12:34:27.944 INFO:teuthology.orchestra.run.vpm133:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-08T12:34:33.418 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4] out_osds: [3, 0] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:33.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:33.418 INFO:tasks.thrashosds.thrasher:Growing pool metadata 2014-08-08T12:34:33.418 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:34:35.407 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:34:35.837 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T12:34:35.838 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pg_num 294' 2014-08-08T12:34:36.957 INFO:teuthology.orchestra.run.vpm132.stderr:set pool 1 pg_num to 294 2014-08-08T12:34:41.998 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2, 4] out_osds: [3, 0] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:41.999 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:41.999 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 5, 2, 4] 2014-08-08T12:34:41.999 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-08-08T12:34:46.255 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.2. 2014-08-08T12:34:51.309 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:51.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:51.309 INFO:tasks.thrashosds.thrasher:Growing pool metadata 2014-08-08T12:34:51.309 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:34:52.164 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:34:57.356 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:34:57.356 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:34:57.356 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-08T12:34:57.356 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T12:34:57.356 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T12:34:57.356 INFO:teuthology.orchestra.run.vpm133:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-08T12:35:02.818 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:02.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:02.818 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.464394354979 2014-08-08T12:35:02.818 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 0.464394354979' 2014-08-08T12:35:04.767 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.5 to 0.464394 (8304342) 2014-08-08T12:35:09.884 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:09.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:09.884 INFO:tasks.thrashosds.thrasher:fixing pg num pool metadata 2014-08-08T12:35:09.885 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:35:10.637 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:35:15.820 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:15.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:15.820 INFO:tasks.thrashosds.thrasher:fixing pg num pool metadata 2014-08-08T12:35:15.820 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:35:17.565 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:35:17.797 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 294' 2014-08-08T12:35:19.823 INFO:teuthology.orchestra.run.vpm132.stderr:set pool 1 pgp_num to 294 2014-08-08T12:35:24.858 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:24.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:24.859 INFO:tasks.thrashosds.thrasher:fixing pg num pool metadata 2014-08-08T12:35:24.859 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:35:25.690 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:35:25.926 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 294' 2014-08-08T12:35:27.595 INFO:teuthology.orchestra.run.vpm132.stderr:set pool 1 pgp_num to 294 2014-08-08T12:35:32.639 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:32.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:32.640 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.70093509366 2014-08-08T12:35:32.640 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 0.70093509366' 2014-08-08T12:35:33.968 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.4 to 0.700935 (8459362) 2014-08-08T12:35:39.007 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:39.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:39.007 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T12:35:39.007 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T12:35:39.007 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T12:35:39.008 INFO:teuthology.orchestra.run.vpm132:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T12:35:44.400 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:44.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:44.400 INFO:tasks.thrashosds.thrasher:Growing pool data 2014-08-08T12:35:44.400 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:35:48.388 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:35:48.561 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T12:35:48.562 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 294' 2014-08-08T12:35:50.980 INFO:teuthology.orchestra.run.vpm132.stderr:set pool 0 pg_num to 294 2014-08-08T12:35:56.020 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4] out_osds: [3, 0, 2] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:35:56.020 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:35:56.020 INFO:tasks.thrashosds.thrasher:Adding osd 2 2014-08-08T12:35:56.021 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-08-08T12:36:00.439 INFO:teuthology.orchestra.run.vpm132.stderr:marked in osd.2. 2014-08-08T12:36:00.488 INFO:tasks.thrashosds.thrasher:Added osd 2 2014-08-08T12:36:05.488 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4, 2] out_osds: [3, 0] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:05.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:05.489 INFO:tasks.thrashosds.thrasher:Adding osd 0 2014-08-08T12:36:05.489 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 0' 2014-08-08T12:36:07.319 INFO:teuthology.orchestra.run.vpm132.stderr:marked in osd.0. 2014-08-08T12:36:07.372 INFO:tasks.thrashosds.thrasher:Added osd 0 2014-08-08T12:36:12.372 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4, 2, 0] out_osds: [3] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:12.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:12.373 INFO:tasks.thrashosds.thrasher:Adding osd 3 2014-08-08T12:36:12.373 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 3' 2014-08-08T12:36:15.419 INFO:teuthology.orchestra.run.vpm132.stderr:marked in osd.3. 2014-08-08T12:36:15.465 INFO:tasks.thrashosds.thrasher:Added osd 3 2014-08-08T12:36:20.465 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4, 2, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:20.466 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:20.466 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-08T12:36:20.466 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:36:21.617 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:36:26.854 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 4, 2, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:26.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:26.854 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 5, 4, 2, 0, 3] 2014-08-08T12:36:26.855 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 1' 2014-08-08T12:36:28.736 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.1. 2014-08-08T12:36:33.786 INFO:tasks.thrashosds.thrasher:in_osds: [5, 4, 2, 0, 3] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:33.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:33.787 INFO:tasks.thrashosds.thrasher:Adding osd 1 2014-08-08T12:36:33.787 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1' 2014-08-08T12:36:35.967 INFO:teuthology.orchestra.run.vpm132.stderr:marked in osd.1. 2014-08-08T12:36:36.052 INFO:tasks.thrashosds.thrasher:Added osd 1 2014-08-08T12:36:41.052 INFO:tasks.thrashosds.thrasher:in_osds: [5, 4, 2, 0, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:41.052 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:41.053 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2014-08-08T12:36:41.053 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T12:36:41.053 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T12:36:41.053 INFO:teuthology.orchestra.run.vpm132:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-08-08T12:36:48.420 INFO:tasks.thrashosds.thrasher:in_osds: [5, 4, 2, 0, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:48.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:36:48.421 INFO:tasks.thrashosds.thrasher:Growing pool data 2014-08-08T12:36:48.421 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:36:50.280 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:36:50.607 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T12:36:50.607 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 304' 2014-08-08T12:36:52.808 INFO:teuthology.orchestra.run.vpm132.stderr:set pool 0 pg_num to 304 2014-08-08T12:36:57.865 INFO:tasks.thrashosds.thrasher:in_osds: [5, 4, 2, 0, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:36:57.865 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T12:36:59.147 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T12:36:59.200 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-08T12:37:01.735 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.4 to 1 (8655362) 2014-08-08T12:37:01.782 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T12:37:04.975 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T12:37:05.027 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-08T12:37:06.557 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.0 to 1 (8655362) 2014-08-08T12:37:06.599 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T12:37:15.632 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T12:37:15.670 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T12:37:18.569 INFO:teuthology.orchestra.run.vpm132.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T12:37:18.614 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T12:37:18.615 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:19.699 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:20.645 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:21.792 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:22.085 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:22.839 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:52.710609', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 76073020, u'health': u'HEALTH_OK', u'kb_used': 22880548, u'store_stats': {u'bytes_total': 58843566, u'bytes_log': 7896035, u'last_updated': u'0.000000', u'bytes_misc': 50947531, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:25.122366', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73169836, u'health': u'HEALTH_OK', u'kb_used': 25783732, u'store_stats': {u'bytes_total': 53592674, u'bytes_log': 2620277, u'last_updated': u'0.000000', u'bytes_misc': 50972397, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'942 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'354 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'354 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'recovering_bytes_per_sec': 126023, u'num_pgs': 1146, u'recovering_keys_per_sec': 168, u'data_bytes': 679071, u'bytes_used': 1204682752, u'recovering_objects_per_sec': 9, u'op_per_sec': 38, u'version': 6285, u'pgs_by_state': [{u'count': 21, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 1, u'state_name': u'active'}, {u'count': 862, u'state_name': u'peering'}, {u'count': 181, u'state_name': u'active+clean'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'read_bytes_sec': 15999, u'write_bytes_sec': 46187, u'bytes_avail': 1286656360448}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2972, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:22.840 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:23.663 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T12:37:23.663 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:24.714 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:27.896 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:28.530 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:28.960 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:29.494 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:52.710609', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 76073020, u'health': u'HEALTH_OK', u'kb_used': 22880548, u'store_stats': {u'bytes_total': 58843566, u'bytes_log': 7896035, u'last_updated': u'0.000000', u'bytes_misc': 50947531, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'910 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'355 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'355 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 679071, u'bytes_used': 1205186560, u'version': 6289, u'pgs_by_state': [{u'count': 43, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 9, u'state_name': u'active'}, {u'count': 830, u'state_name': u'peering'}, {u'count': 183, u'state_name': u'active+clean'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286655856640}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2972, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:29.495 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:29.710 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:37:29.711 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:30.029 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:33.234 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:34.072 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:34.449 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:36.154 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:52.710609', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 76073020, u'health': u'HEALTH_OK', u'kb_used': 22880548, u'store_stats': {u'bytes_total': 58843566, u'bytes_log': 7896035, u'last_updated': u'0.000000', u'bytes_misc': 50947531, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'895 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'354 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'354 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 679071, u'bytes_used': 1204940800, u'version': 6293, u'pgs_by_state': [{u'count': 21, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 815, u'state_name': u'peering'}, {u'count': 229, u'state_name': u'active+clean'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286656102400}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2972, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:36.155 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:37.023 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:37:37.023 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:38.400 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:41.592 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:42.674 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:43.015 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:43.756 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:52.710609', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 76073020, u'health': u'HEALTH_OK', u'kb_used': 22880548, u'store_stats': {u'bytes_total': 58843566, u'bytes_log': 7896035, u'last_updated': u'0.000000', u'bytes_misc': 50947531, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'868 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'353 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'353 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6/398 objects degraded (1.508%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'degraded_objects': 6, u'num_pgs': 1146, u'data_bytes': 679071, u'degraded_total': 398, u'bytes_used': 1210642432, u'version': 6298, u'pgs_by_state': [{u'count': 19, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 1, u'state_name': u'active'}, {u'count': 788, u'state_name': u'peering'}, {u'count': 257, u'state_name': u'active+clean'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'degraded_ratio': u'1.508', u'bytes_avail': 1286650400768}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2972, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:43.757 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:44.361 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:37:44.362 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:45.110 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:48.264 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:49.124 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:49.867 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:50.652 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:36:52.710609', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 76073020, u'health': u'HEALTH_OK', u'kb_used': 22880548, u'store_stats': {u'bytes_total': 58843566, u'bytes_log': 7896035, u'last_updated': u'0.000000', u'bytes_misc': 50947531, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'819 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'347 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'347 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'recovering_bytes_per_sec': 0, u'num_pgs': 1146, u'recovering_keys_per_sec': 0, u'data_bytes': 679071, u'bytes_used': 1211686912, u'recovering_objects_per_sec': 1, u'version': 6303, u'pgs_by_state': [{u'count': 19, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 739, u'state_name': u'peering'}, {u'count': 307, u'state_name': u'active+clean'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286649356288}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2972, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:50.652 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:51.633 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:37:51.634 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:53.215 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:56.506 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:37:57.472 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:37:57.754 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:58.751 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'789 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'236 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'236 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 754963, u'bytes_used': 1214488576, u'version': 6310, u'pgs_by_state': [{u'count': 25, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 3, u'state_name': u'active'}, {u'count': 703, u'state_name': u'peering'}, {u'count': 327, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 86, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286646554624}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2974, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:37:58.751 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:37:59.732 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:37:59.733 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:00.528 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:03.676 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:04.937 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:05.205 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:05.910 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'728 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'227 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'227 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 754997, u'bytes_used': 1210605568, u'version': 6316, u'pgs_by_state': [{u'count': 46, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 643, u'state_name': u'peering'}, {u'count': 370, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 85, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286650437632}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2974, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:05.910 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:10.369 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:38:10.369 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:11.558 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:14.808 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:15.826 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:16.038 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:16.885 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:37:18.602101', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73023692, u'health': u'HEALTH_OK', u'kb_used': 25929876, u'store_stats': {u'bytes_total': 52815383, u'bytes_log': 1496924, u'last_updated': u'0.000000', u'bytes_misc': 51318459, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'595 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'180 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'180 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 754997, u'bytes_used': 1217921024, u'version': 6323, u'pgs_by_state': [{u'count': 45, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 511, u'state_name': u'peering'}, {u'count': 504, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 84, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286643122176}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2974, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:16.885 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:17.715 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:38:17.715 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:18.704 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:21.880 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:23.143 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:23.435 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:24.449 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:25.122843', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 73004688, u'health': u'HEALTH_OK', u'kb_used': 25948880, u'store_stats': {u'bytes_total': 54708767, u'bytes_log': 3393946, u'last_updated': u'0.000000', u'bytes_misc': 51314821, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'528 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'136 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'136 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 755223, u'bytes_used': 1212743680, u'version': 6329, u'pgs_by_state': [{u'count': 45, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 446, u'state_name': u'peering'}, {u'count': 572, u'state_name': u'active+clean'}, {u'count': 82, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286648299520}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2974, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:24.450 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:25.415 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:38:25.416 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:28.468 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:31.610 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:32.498 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:32.759 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:33.619 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:25.123410', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72839240, u'health': u'HEALTH_OK', u'kb_used': 26114328, u'store_stats': {u'bytes_total': 57730477, u'bytes_log': 5179361, u'last_updated': u'0.000000', u'bytes_misc': 52551116, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'439 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'110 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'110 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'recovering_bytes_per_sec': 74, u'num_pgs': 1146, u'recovering_keys_per_sec': 0, u'data_bytes': 757425, u'bytes_used': 1217888256, u'recovering_objects_per_sec': 0, u'version': 6336, u'pgs_by_state': [{u'count': 45, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 359, u'state_name': u'peering'}, {u'count': 660, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 80, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286643154944}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2974, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:33.619 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:34.518 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T12:38:34.518 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:35.809 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:39.032 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:39.724 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:39.958 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:41.271 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:37:52.711104', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75968476, u'health': u'HEALTH_OK', u'kb_used': 22985092, u'store_stats': {u'bytes_total': 57086382, u'bytes_log': 1020141, u'last_updated': u'0.000000', u'bytes_misc': 56066241, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:25.123410', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72839240, u'health': u'HEALTH_OK', u'kb_used': 26114328, u'store_stats': {u'bytes_total': 57730477, u'bytes_log': 5179361, u'last_updated': u'0.000000', u'bytes_misc': 52551116, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'415 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'104 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'104 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 757425, u'bytes_used': 1221898240, u'version': 6342, u'pgs_by_state': [{u'count': 45, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 338, u'state_name': u'peering'}, {u'count': 684, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 77, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286639144960}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2976, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:41.271 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:45.520 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:38:45.520 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:47.007 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:50.107 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:52.389 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:52.707 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:53.680 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:52.711669', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75893384, u'health': u'HEALTH_OK', u'kb_used': 23060184, u'store_stats': {u'bytes_total': 50418504, u'bytes_log': 2502678, u'last_updated': u'0.000000', u'bytes_misc': 47915826, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:25.123410', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72839240, u'health': u'HEALTH_OK', u'kb_used': 26114328, u'store_stats': {u'bytes_total': 57730477, u'bytes_log': 5179361, u'last_updated': u'0.000000', u'bytes_misc': 52551116, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'274 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'125 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'125 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 757425, u'bytes_used': 1212854272, u'version': 6350, u'pgs_by_state': [{u'count': 174, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 202, u'state_name': u'peering'}, {u'count': 695, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 1, u'state_name': u'remapped'}, {u'count': 72, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286648188928}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2978, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:38:53.680 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:38:54.525 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T12:38:54.525 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:38:55.909 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:38:59.064 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:39:00.245 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:39:00.557 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:39:01.650 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:52.711669', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75893384, u'health': u'HEALTH_OK', u'kb_used': 23060184, u'store_stats': {u'bytes_total': 50418504, u'bytes_log': 2502678, u'last_updated': u'0.000000', u'bytes_misc': 47915826, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:25.123410', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72839240, u'health': u'HEALTH_OK', u'kb_used': 26114328, u'store_stats': {u'bytes_total': 57730477, u'bytes_log': 5179361, u'last_updated': u'0.000000', u'bytes_misc': 52551116, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'251 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'128 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'129 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 757425, u'bytes_used': 1215426560, u'version': 6358, u'pgs_by_state': [{u'count': 188, u'state_name': u'inactive'}, {u'count': 179, u'state_name': u'peering'}, {u'count': 707, u'state_name': u'active+clean'}, {u'count': 72, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286645616640}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2981, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:39:01.651 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:39:02.477 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:39:02.477 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:39:03.088 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:39:06.243 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:39:07.265 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:39:07.702 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:39:09.213 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 16, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.139.246:6789/0'}, {u'name': u'c', u'rank': 1, u'addr': u'10.214.140.182:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.246:6790/0'}], u'modified': u'2014-08-08 16:51:44.007538', u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585', u'created': u'2014-08-08 16:51:44.007538'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 16, u'round': 44, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.035035', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.005442', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 19:38:18.604890', u'name': u'a', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72857700, u'health': u'HEALTH_OK', u'kb_used': 26095868, u'store_stats': {u'bytes_total': 55758075, u'bytes_log': 3203275, u'last_updated': u'0.000000', u'bytes_misc': 52554800, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:52.711669', u'name': u'c', u'avail_percent': 73, u'kb_total': 103178528, u'kb_avail': 75893384, u'health': u'HEALTH_OK', u'kb_used': 23060184, u'store_stats': {u'bytes_total': 50418504, u'bytes_log': 2502678, u'last_updated': u'0.000000', u'bytes_misc': 47915826, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 19:38:25.123410', u'name': u'b', u'avail_percent': 70, u'kb_total': 103178528, u'kb_avail': 72839240, u'health': u'HEALTH_OK', u'kb_used': 26114328, u'store_stats': {u'bytes_total': 57730477, u'bytes_log': 5179361, u'last_updated': u'0.000000', u'bytes_misc': 52551116, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'196 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'124 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'125 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 274'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 404 > pgp_num 384'}]}, u'pgmap': {u'bytes_total': 1287861043200, u'num_pgs': 1146, u'data_bytes': 832597, u'bytes_used': 1215836160, u'version': 6363, u'pgs_by_state': [{u'count': 225, u'state_name': u'inactive'}, {u'count': 130, u'state_name': u'peering'}, {u'count': 725, u'state_name': u'active+clean'}, {u'count': 66, u'state_name': u'remapped+peering'}], u'bytes_avail': 1286645207040}, u'quorum_names': [u'a', u'c', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2985, u'num_in_osds': 6}}, u'fsid': u'199e05e1-c5cf-4781-858f-2eaf9f969585'} 2014-08-08T12:39:09.213 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T12:39:10.216 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T12:39:10.216 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:39:11.346 INFO:teuthology.orchestra.run.vpm132.stderr:dumped all in format json 2014-08-08T12:39:14.636 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T12:39:15.291 INFO:teuthology.orchestra.run.vpm134.stderr:testBadHeaders (test.functional.tests.TestFileUTF8) ... ERROR ---------------------------------------------------------------------- 2014-08-08T10:48:37.556 INFO:teuthology.orchestra.run.vpm137.stderr:testContainerListing (test.functional.tests.TestAccount) ... ok 2014-08-08T10:48:37.747 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:37.815 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:38.031 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:48:13.452609', u'name': u'a', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89775840, u'health': u'HEALTH_OK', u'kb_used': 9177728, u'store_stats': {u'bytes_total': 23302015, u'bytes_log': 3880155, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 198/1032 objects degraded (19.186%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'recovering_bytes_per_sec': 8889122, u'degraded_objects': 198, u'num_pgs': 258, u'recovering_keys_per_sec': 0, u'data_bytes': 813859240, u'degraded_total': 1032, u'bytes_used': 1579646976, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 1664322, u'op_per_sec': 20, u'version': 1651, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 237, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'read_bytes_sec': 2860811, u'degraded_ratio': u'19.186', u'bytes_avail': 427707367424}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 4, u'epoch': 617, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:48:38.032 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:38.308 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:48:38.308 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:38.732 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:38.864 INFO:tasks.rados.rados.0.vpm137.stdout:3337: done (15 left) 2014-08-08T10:48:38.864 INFO:tasks.rados.rados.0.vpm137.stdout:3354: done (14 left) 2014-08-08T10:48:38.864 INFO:tasks.rados.rados.0.vpm137.stdout:3355: delete oid 396 current snap is 0 2014-08-08T10:48:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3340: finishing write tid 1 to vpm1375878-18 2014-08-08T10:48:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3340: finishing write tid 2 to vpm1375878-18 2014-08-08T10:48:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3340: finishing write tid 4 to vpm1375878-18 2014-08-08T10:48:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3344: finishing write tid 1 to vpm1375878-460 2014-08-08T10:48:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3344: finishing write tid 3 to vpm1375878-460 2014-08-08T10:48:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3347: finishing write tid 1 to vpm1375878-221 2014-08-08T10:48:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3347: finishing write tid 2 to vpm1375878-221 2014-08-08T10:48:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3347: finishing write tid 4 to vpm1375878-221 2014-08-08T10:48:38.867 INFO:tasks.rados.rados.0.vpm137.stdout:3348: finishing write tid 1 to vpm1375878-280 2014-08-08T10:48:38.868 INFO:tasks.rados.rados.0.vpm137.stdout:3348: finishing write tid 2 to vpm1375878-280 2014-08-08T10:48:38.868 INFO:tasks.rados.rados.0.vpm137.stdout:3346: finishing write tid 1 to vpm1375878-498 2014-08-08T10:48:38.868 INFO:tasks.rados.rados.0.vpm137.stdout:3346: finishing write tid 2 to vpm1375878-498 2014-08-08T10:48:38.868 INFO:tasks.rados.rados.0.vpm137.stdout:3346: finishing write tid 3 to vpm1375878-498 2014-08-08T10:48:38.868 INFO:tasks.rados.rados.0.vpm137.stdout:3355: done (14 left) 2014-08-08T10:48:38.869 INFO:tasks.rados.rados.0.vpm137.stdout:3356: read oid 173 snap -1 2014-08-08T10:48:38.869 INFO:tasks.rados.rados.0.vpm137.stdout:3357: write oid 16 current snap is 0 2014-08-08T10:48:38.869 INFO:tasks.rados.rados.0.vpm137.stdout:3357: seq_num 2078 ranges {578468=1} 2014-08-08T10:48:38.872 INFO:tasks.rados.rados.0.vpm137.stdout:3357: writing vpm1375878-16 from 578468 to 578469 tid 1 2014-08-08T10:48:38.873 INFO:tasks.rados.rados.0.vpm137.stdout:3338: finishing write tid 1 to vpm1375878-209 2014-08-08T10:48:38.873 INFO:tasks.rados.rados.0.vpm137.stdout:3338: finishing write tid 2 to vpm1375878-209 2014-08-08T10:48:38.873 INFO:tasks.rados.rados.0.vpm137.stdout:3338: finishing write tid 3 to vpm1375878-209 2014-08-08T10:48:38.874 INFO:tasks.rados.rados.0.vpm137.stdout:3338: finishing write tid 5 to vpm1375878-209 2014-08-08T10:48:38.874 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 1 to vpm1375878-10 2014-08-08T10:48:38.874 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 2 to vpm1375878-10 2014-08-08T10:48:38.875 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 3 to vpm1375878-10 2014-08-08T10:48:38.875 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 4 to vpm1375878-10 2014-08-08T10:48:38.875 INFO:tasks.rados.rados.0.vpm137.stdout:3348: finishing write tid 3 to vpm1375878-280 2014-08-08T10:48:38.875 INFO:tasks.rados.rados.0.vpm137.stdout:3348: finishing write tid 5 to vpm1375878-280 2014-08-08T10:48:38.878 INFO:tasks.rados.rados.0.vpm137.stdout:3346: finishing write tid 5 to vpm1375878-498 2014-08-08T10:48:38.878 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 6 to vpm1375878-10 2014-08-08T10:48:38.879 INFO:tasks.rados.rados.0.vpm137.stdout:3339: finishing write tid 7 to vpm1375878-10 2014-08-08T10:48:38.879 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 10 v 464 (ObjNum 2069 snap 0 seq_num 2069) dirty exists 2014-08-08T10:48:38.879 INFO:tasks.rados.rados.0.vpm137.stdout:3338: finishing write tid 6 to vpm1375878-209 2014-08-08T10:48:38.879 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 209 v 459 (ObjNum 2068 snap 0 seq_num 2068) dirty exists 2014-08-08T10:48:38.879 INFO:tasks.rados.rados.0.vpm137.stdout:3342: expect (ObjNum 1438 snap 0 seq_num 1438) 2014-08-08T10:48:40.638 INFO:tasks.rados.rados.0.vpm137.stdout:3338: done (15 left) 2014-08-08T10:48:40.638 INFO:tasks.rados.rados.0.vpm137.stdout:3339: done (14 left) 2014-08-08T10:48:40.639 INFO:tasks.rados.rados.0.vpm137.stdout:3342: done (13 left) 2014-08-08T10:48:40.639 INFO:tasks.rados.rados.0.vpm137.stdout:3358: write oid 145 current snap is 0 2014-08-08T10:48:40.639 INFO:tasks.rados.rados.0.vpm137.stdout:3358: seq_num 2079 ranges {638627=790669,1838626=1} 2014-08-08T10:48:40.700 INFO:tasks.rados.rados.0.vpm137.stdout:3358: writing vpm1375878-145 from 638627 to 1429296 tid 1 2014-08-08T10:48:40.727 INFO:tasks.rados.rados.0.vpm137.stdout:3358: writing vpm1375878-145 from 1838626 to 1838627 tid 2 2014-08-08T10:48:40.728 INFO:tasks.rados.rados.0.vpm137.stdout:3344: finishing write tid 4 to vpm1375878-460 2014-08-08T10:48:40.728 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 460 v 326 (ObjNum 2071 snap 0 seq_num 2071) dirty exists 2014-08-08T10:48:40.729 INFO:tasks.rados.rados.0.vpm137.stdout:3340: finishing write tid 5 to vpm1375878-18 2014-08-08T10:48:40.729 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 18 v 373 (ObjNum 2070 snap 0 seq_num 2070) dirty exists 2014-08-08T10:48:40.729 INFO:tasks.rados.rados.0.vpm137.stdout:3347: finishing write tid 5 to vpm1375878-221 2014-08-08T10:48:40.730 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 221 v 340 (ObjNum 2074 snap 0 seq_num 2074) dirty exists 2014-08-08T10:48:40.730 INFO:tasks.rados.rados.0.vpm137.stdout:3345: finishing write tid 1 to vpm1375878-496 2014-08-08T10:48:40.730 INFO:tasks.rados.rados.0.vpm137.stdout:3345: finishing write tid 2 to vpm1375878-496 2014-08-08T10:48:40.730 INFO:tasks.rados.rados.0.vpm137.stdout:3345: finishing write tid 4 to vpm1375878-496 2014-08-08T10:48:40.731 INFO:tasks.rados.rados.0.vpm137.stdout:3346: finishing write tid 6 to vpm1375878-498 2014-08-08T10:48:40.731 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 498 v 565 (ObjNum 2073 snap 0 seq_num 2073) dirty exists 2014-08-08T10:48:40.736 INFO:tasks.rados.rados.0.vpm137.stdout:3348: finishing write tid 6 to vpm1375878-280 2014-08-08T10:48:40.736 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 280 v 396 (ObjNum 2075 snap 0 seq_num 2075) dirty exists 2014-08-08T10:48:40.736 INFO:tasks.rados.rados.0.vpm137.stdout:3352: expect (ObjNum 2043 snap 0 seq_num 2043) 2014-08-08T10:48:40.802 INFO:tasks.rados.rados.0.vpm137.stdout:3340: done (13 left) 2014-08-08T10:48:40.803 INFO:tasks.rados.rados.0.vpm137.stdout:3344: done (12 left) 2014-08-08T10:48:40.803 INFO:tasks.rados.rados.0.vpm137.stdout:3346: done (11 left) 2014-08-08T10:48:40.803 INFO:tasks.rados.rados.0.vpm137.stdout:3347: done (10 left) 2014-08-08T10:48:40.803 INFO:tasks.rados.rados.0.vpm137.stdout:3348: done (9 left) 2014-08-08T10:48:40.804 INFO:tasks.rados.rados.0.vpm137.stdout:3349: done (8 left) 2014-08-08T10:48:40.804 INFO:tasks.rados.rados.0.vpm137.stdout:3352: done (7 left) 2014-08-08T10:48:40.804 INFO:tasks.rados.rados.0.vpm137.stdout:3359: read oid 400 snap -1 2014-08-08T10:48:40.805 INFO:tasks.rados.rados.0.vpm137.stdout:3360: read oid 7 snap -1 2014-08-08T10:48:40.805 INFO:tasks.rados.rados.0.vpm137.stdout:3361: write oid 111 current snap is 0 2014-08-08T10:48:40.805 INFO:tasks.rados.rados.0.vpm137.stdout:3361: seq_num 2080 ranges {733857=513897,1533856=1} 2014-08-08T10:48:40.840 INFO:tasks.rados.rados.0.vpm137.stdout:3361: writing vpm1375878-111 from 733857 to 1247754 tid 1 2014-08-08T10:48:40.847 INFO:tasks.rados.rados.0.vpm137.stdout:3361: writing vpm1375878-111 from 1533856 to 1533857 tid 2 2014-08-08T10:48:40.848 INFO:tasks.rados.rados.0.vpm137.stdout:3350: expect (ObjNum 2050 snap 0 seq_num 2050) 2014-08-08T10:48:41.763 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:41.969 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:42.012 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:42.441 INFO:tasks.rados.rados.0.vpm137.stdout:3350: done (9 left) 2014-08-08T10:48:42.441 INFO:tasks.rados.rados.0.vpm137.stdout:3362: read oid 307 snap -1 2014-08-08T10:48:42.442 INFO:tasks.rados.rados.0.vpm137.stdout:3363: write oid 95 current snap is 0 2014-08-08T10:48:42.442 INFO:tasks.rados.rados.0.vpm137.stdout:3363: seq_num 2081 ranges {678687=734278,2084084=194603} 2014-08-08T10:48:42.476 INFO:tasks.rados.rados.0.vpm137.stdout:3363: writing vpm1375878-95 from 678687 to 1412965 tid 1 2014-08-08T10:48:42.494 INFO:tasks.rados.rados.0.vpm137.stdout:3363: writing vpm1375878-95 from 2084084 to 2278687 tid 2 2014-08-08T10:48:42.494 INFO:tasks.rados.rados.0.vpm137.stdout:3353: expect (ObjNum 1419 snap 0 seq_num 1419) 2014-08-08T10:48:43.359 INFO:tasks.rados.rados.0.vpm137.stdout:3353: done (10 left) 2014-08-08T10:48:43.359 INFO:tasks.rados.rados.0.vpm137.stdout:3364: write oid 217 current snap is 0 2014-08-08T10:48:43.360 INFO:tasks.rados.rados.0.vpm137.stdout:3364: seq_num 2082 ranges {622492=742178,2081863=456698,3022491=1} 2014-08-08T10:48:43.387 INFO:tasks.rados.rados.0.vpm137.stdout:3364: writing vpm1375878-217 from 622492 to 1364670 tid 1 2014-08-08T10:48:43.414 INFO:tasks.rados.rados.0.vpm137.stdout:3364: writing vpm1375878-217 from 2081863 to 2538561 tid 2 2014-08-08T10:48:43.419 INFO:tasks.rados.rados.0.vpm137.stdout:3364: writing vpm1375878-217 from 3022491 to 3022492 tid 3 2014-08-08T10:48:43.420 INFO:tasks.rados.rados.0.vpm137.stdout:3351: expect (ObjNum 1634 snap 0 seq_num 1634) 2014-08-08T10:48:45.245 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:48:13.452609', u'name': u'a', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89775840, u'health': u'HEALTH_OK', u'kb_used': 9177728, u'store_stats': {u'bytes_total': 23302015, u'bytes_log': 3880155, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 179/1036 objects degraded (17.278%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'recovering_bytes_per_sec': 7157912, u'degraded_objects': 179, u'num_pgs': 258, u'recovering_keys_per_sec': 0, u'data_bytes': 811478676, u'degraded_total': 1036, u'bytes_used': 1612128256, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 1239833, u'op_per_sec': 26, u'version': 1652, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 237, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'read_bytes_sec': 2748736, u'degraded_ratio': u'17.278', u'bytes_avail': 427674886144}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 4, u'epoch': 617, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:48:45.246 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:45.352 INFO:tasks.rados.rados.0.vpm137.stdout:3351: done (10 left) 2014-08-08T10:48:45.352 INFO:tasks.rados.rados.0.vpm137.stdout:3365: write oid 127 current snap is 0 2014-08-08T10:48:45.353 INFO:tasks.rados.rados.0.vpm137.stdout:3365: seq_num 2083 ranges {435098=715006,1629723=526893,2850788=384310} 2014-08-08T10:48:45.379 INFO:tasks.rados.rados.0.vpm137.stdout:3365: writing vpm1375878-127 from 435098 to 1150104 tid 1 2014-08-08T10:48:45.406 INFO:tasks.rados.rados.0.vpm137.stdout:3365: writing vpm1375878-127 from 1629723 to 2156616 tid 2 2014-08-08T10:48:45.435 INFO:tasks.rados.rados.0.vpm137.stdout:3365: writing vpm1375878-127 from 2850788 to 3235098 tid 3 2014-08-08T10:48:45.438 INFO:tasks.rados.rados.0.vpm137.stdout:3345: finishing write tid 5 to vpm1375878-496 2014-08-08T10:48:45.439 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 496 v 379 (ObjNum 2072 snap 0 seq_num 2072) dirty exists 2014-08-08T10:48:45.439 INFO:tasks.rados.rados.0.vpm137.stdout:3357: finishing write tid 1 to vpm1375878-16 2014-08-08T10:48:45.439 INFO:tasks.rados.rados.0.vpm137.stdout:3357: finishing write tid 3 to vpm1375878-16 2014-08-08T10:48:45.439 INFO:tasks.rados.rados.0.vpm137.stdout:3357: finishing write tid 4 to vpm1375878-16 2014-08-08T10:48:45.440 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 16 v 466 (ObjNum 2078 snap 0 seq_num 2078) dirty exists 2014-08-08T10:48:45.440 INFO:tasks.rados.rados.0.vpm137.stdout:3356: expect (ObjNum 2008 snap 0 seq_num 2008) 2014-08-08T10:48:45.642 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:48:45.643 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:45.844 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:46.800 INFO:tasks.rados.rados.0.vpm137.stdout:3345: done (10 left) 2014-08-08T10:48:46.800 INFO:tasks.rados.rados.0.vpm137.stdout:3356: done (9 left) 2014-08-08T10:48:46.800 INFO:tasks.rados.rados.0.vpm137.stdout:3357: done (8 left) 2014-08-08T10:48:46.800 INFO:tasks.rados.rados.0.vpm137.stdout:3366: read oid 221 snap -1 2014-08-08T10:48:46.801 INFO:tasks.rados.rados.0.vpm137.stdout:3367: write oid 479 current snap is 0 2014-08-08T10:48:46.801 INFO:tasks.rados.rados.0.vpm137.stdout:3367: seq_num 2084 ranges {779928=438235,1992432=597089,3301615=678313} 2014-08-08T10:48:46.822 INFO:tasks.rados.rados.0.vpm137.stdout:3367: writing vpm1375878-479 from 779928 to 1218163 tid 1 2014-08-08T10:48:46.851 INFO:tasks.rados.rados.0.vpm137.stdout:3367: writing vpm1375878-479 from 1992432 to 2589521 tid 2 2014-08-08T10:48:46.894 INFO:tasks.rados.rados.0.vpm137.stdout:3367: writing vpm1375878-479 from 3301615 to 3979928 tid 3 2014-08-08T10:48:46.900 INFO:tasks.rados.rados.0.vpm137.stdout:3360: expect (ObjNum 810 snap 0 seq_num 810) 2014-08-08T10:48:46.970 INFO:tasks.rados.rados.0.vpm137.stdout:3360: done (9 left) 2014-08-08T10:48:46.970 INFO:tasks.rados.rados.0.vpm137.stdout:3368: read oid 143 snap -1 2014-08-08T10:48:46.970 INFO:tasks.rados.rados.0.vpm137.stdout:3369: write oid 309 current snap is 0 2014-08-08T10:48:46.971 INFO:tasks.rados.rados.0.vpm137.stdout:3369: seq_num 2085 ranges {475158=410039,1305540=418708,2453240=526559,3388572=286586} 2014-08-08T10:48:47.005 INFO:tasks.rados.rados.0.vpm137.stdout:3369: writing vpm1375878-309 from 475158 to 885197 tid 1 2014-08-08T10:48:47.034 INFO:tasks.rados.rados.0.vpm137.stdout:3369: writing vpm1375878-309 from 1305540 to 1724248 tid 2 2014-08-08T10:48:47.073 INFO:tasks.rados.rados.0.vpm137.stdout:3369: writing vpm1375878-309 from 2453240 to 2979799 tid 3 2014-08-08T10:48:47.097 INFO:tasks.rados.rados.0.vpm137.stdout:3369: writing vpm1375878-309 from 3388572 to 3675158 tid 4 2014-08-08T10:48:47.099 INFO:tasks.rados.rados.0.vpm137.stdout:3358: finishing write tid 1 to vpm1375878-145 2014-08-08T10:48:47.100 INFO:tasks.rados.rados.0.vpm137.stdout:3358: finishing write tid 2 to vpm1375878-145 2014-08-08T10:48:47.100 INFO:tasks.rados.rados.0.vpm137.stdout:3358: finishing write tid 4 to vpm1375878-145 2014-08-08T10:48:47.100 INFO:tasks.rados.rados.0.vpm137.stdout:3361: finishing write tid 1 to vpm1375878-111 2014-08-08T10:48:47.101 INFO:tasks.rados.rados.0.vpm137.stdout:3361: finishing write tid 2 to vpm1375878-111 2014-08-08T10:48:47.101 INFO:tasks.rados.rados.0.vpm137.stdout:3361: finishing write tid 4 to vpm1375878-111 2014-08-08T10:48:47.104 INFO:tasks.rados.rados.0.vpm137.stdout:3359: expect (ObjNum 1519 snap 0 seq_num 1519) 2014-08-08T10:48:48.605 INFO:tasks.rados.rados.0.vpm137.stdout:3359: done (10 left) 2014-08-08T10:48:48.607 INFO:tasks.rados.rados.0.vpm137.stdout:3370: read oid 453 snap -1 2014-08-08T10:48:48.607 INFO:tasks.rados.rados.0.vpm137.stdout:3371: read oid 351 snap -1 2014-08-08T10:48:48.608 INFO:tasks.rados.rados.0.vpm137.stdout:3372: read oid 248 snap -1 2014-08-08T10:48:48.608 INFO:tasks.rados.rados.0.vpm137.stdout:3373: write oid 117 current snap is 0 2014-08-08T10:48:48.608 INFO:tasks.rados.rados.0.vpm137.stdout:3373: seq_num 2086 ranges {419986=1} 2014-08-08T10:48:48.611 INFO:tasks.rados.rados.0.vpm137.stdout:3373: writing vpm1375878-117 from 419986 to 419987 tid 1 2014-08-08T10:48:48.612 INFO:tasks.rados.rados.0.vpm137.stdout:3358: finishing write tid 5 to vpm1375878-145 2014-08-08T10:48:48.613 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 145 v 399 (ObjNum 2079 snap 0 seq_num 2079) dirty exists 2014-08-08T10:48:48.613 INFO:tasks.rados.rados.0.vpm137.stdout:3361: finishing write tid 5 to vpm1375878-111 2014-08-08T10:48:48.613 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 111 v 568 (ObjNum 2080 snap 0 seq_num 2080) dirty exists 2014-08-08T10:48:48.613 INFO:tasks.rados.rados.0.vpm137.stdout:3362: expect (ObjNum 616 snap 0 seq_num 616) 2014-08-08T10:48:48.816 INFO:tasks.rados.rados.0.vpm137.stdout:3358: done (13 left) 2014-08-08T10:48:48.816 INFO:tasks.rados.rados.0.vpm137.stdout:3361: done (12 left) 2014-08-08T10:48:48.817 INFO:tasks.rados.rados.0.vpm137.stdout:3362: done (11 left) 2014-08-08T10:48:48.817 INFO:tasks.rados.rados.0.vpm137.stdout:3374: write oid 454 current snap is 0 2014-08-08T10:48:48.817 INFO:tasks.rados.rados.0.vpm137.stdout:3374: seq_num 2087 ranges {762769=400000} 2014-08-08T10:48:48.835 INFO:tasks.rados.rados.0.vpm137.stdout:3374: writing vpm1375878-454 from 762769 to 1162769 tid 1 2014-08-08T10:48:48.837 INFO:tasks.rados.rados.0.vpm137.stdout:3365: finishing write tid 1 to vpm1375878-127 2014-08-08T10:48:48.837 INFO:tasks.rados.rados.0.vpm137.stdout:3365: finishing write tid 2 to vpm1375878-127 2014-08-08T10:48:48.837 INFO:tasks.rados.rados.0.vpm137.stdout:3365: finishing write tid 3 to vpm1375878-127 2014-08-08T10:48:48.837 INFO:tasks.rados.rados.0.vpm137.stdout:3365: finishing write tid 5 to vpm1375878-127 2014-08-08T10:48:48.837 INFO:tasks.rados.rados.0.vpm137.stdout:3365: finishing write tid 6 to vpm1375878-127 2014-08-08T10:48:48.838 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 127 v 385 (ObjNum 2083 snap 0 seq_num 2083) dirty exists 2014-08-08T10:48:48.838 INFO:tasks.rados.rados.0.vpm137.stdout:3366: expect (ObjNum 2074 snap 0 seq_num 2074) 2014-08-08T10:48:48.874 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:49.178 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:49.238 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:49.781 INFO:tasks.rados.rados.0.vpm137.stdout:3365: done (11 left) 2014-08-08T10:48:49.782 INFO:tasks.rados.rados.0.vpm137.stdout:3366: done (10 left) 2014-08-08T10:48:49.782 INFO:tasks.rados.rados.0.vpm137.stdout:3375: read oid 289 snap -1 2014-08-08T10:48:49.782 INFO:tasks.rados.rados.0.vpm137.stdout:3376: write oid 214 current snap is 0 2014-08-08T10:48:49.783 INFO:tasks.rados.rados.0.vpm137.stdout:3376: seq_num 2088 ranges {575375=512972,1375374=1} 2014-08-08T10:48:49.804 INFO:tasks.rados.rados.0.vpm137.stdout:3376: writing vpm1375878-214 from 575375 to 1088347 tid 1 2014-08-08T10:48:49.809 INFO:tasks.rados.rados.0.vpm137.stdout:3376: writing vpm1375878-214 from 1375374 to 1375375 tid 2 2014-08-08T10:48:49.811 INFO:tasks.rados.rados.0.vpm137.stdout:3368: expect (ObjNum 1857 snap 0 seq_num 1857) 2014-08-08T10:48:49.884 INFO:tasks.rados.rados.0.vpm137.stdout:3368: done (11 left) 2014-08-08T10:48:49.884 INFO:tasks.rados.rados.0.vpm137.stdout:3377: write oid 223 current snap is 0 2014-08-08T10:48:49.885 INFO:tasks.rados.rados.0.vpm137.stdout:3377: seq_num 2089 ranges {520205=637224,1842324=277881} 2014-08-08T10:48:49.911 INFO:tasks.rados.rados.0.vpm137.stdout:3377: writing vpm1375878-223 from 520205 to 1157429 tid 1 2014-08-08T10:48:49.932 INFO:tasks.rados.rados.0.vpm137.stdout:3377: writing vpm1375878-223 from 1842324 to 2120205 tid 2 2014-08-08T10:48:49.934 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 1 to vpm1375878-309 2014-08-08T10:48:49.934 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 2 to vpm1375878-309 2014-08-08T10:48:49.934 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 3 to vpm1375878-309 2014-08-08T10:48:49.934 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 4 to vpm1375878-309 2014-08-08T10:48:49.935 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 6 to vpm1375878-309 2014-08-08T10:48:49.935 INFO:tasks.rados.rados.0.vpm137.stdout:3373: finishing write tid 1 to vpm1375878-117 2014-08-08T10:48:49.935 INFO:tasks.rados.rados.0.vpm137.stdout:3373: finishing write tid 3 to vpm1375878-117 2014-08-08T10:48:49.935 INFO:tasks.rados.rados.0.vpm137.stdout:3364: finishing write tid 1 to vpm1375878-217 2014-08-08T10:48:49.935 INFO:tasks.rados.rados.0.vpm137.stdout:3373: finishing write tid 4 to vpm1375878-117 2014-08-08T10:48:49.936 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 117 v 288 (ObjNum 2086 snap 0 seq_num 2086) dirty exists 2014-08-08T10:48:49.936 INFO:tasks.rados.rados.0.vpm137.stdout:3370: expect (ObjNum 1566 snap 0 seq_num 1566) 2014-08-08T10:48:50.871 INFO:tasks.rados.rados.0.vpm137.stdout:3370: done (11 left) 2014-08-08T10:48:50.872 INFO:tasks.rados.rados.0.vpm137.stdout:3373: done (10 left) 2014-08-08T10:48:50.872 INFO:tasks.rados.rados.0.vpm137.stdout:3378: read oid 361 snap -1 2014-08-08T10:48:50.872 INFO:tasks.rados.rados.0.vpm137.stdout:3379: write oid 85 current snap is 0 2014-08-08T10:48:50.873 INFO:tasks.rados.rados.0.vpm137.stdout:3379: seq_num 2090 ranges {464010=492677,1652584=404756,2488742=375268} 2014-08-08T10:48:50.912 INFO:tasks.rados.rados.0.vpm137.stdout:3379: writing vpm1375878-85 from 464010 to 956687 tid 1 2014-08-08T10:48:50.945 INFO:tasks.rados.rados.0.vpm137.stdout:3379: writing vpm1375878-85 from 1652584 to 2057340 tid 2 2014-08-08T10:48:50.973 INFO:tasks.rados.rados.0.vpm137.stdout:3379: writing vpm1375878-85 from 2488742 to 2864010 tid 3 2014-08-08T10:48:50.977 INFO:tasks.rados.rados.0.vpm137.stdout:3369: finishing write tid 7 to vpm1375878-309 2014-08-08T10:48:50.977 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 309 v 414 (ObjNum 2085 snap 0 seq_num 2085) dirty exists 2014-08-08T10:48:50.978 INFO:tasks.rados.rados.0.vpm137.stdout:3372: expect (ObjNum 1687 snap 0 seq_num 1687) 2014-08-08T10:48:51.722 INFO:tasks.rados.rados.0.vpm137.stdout:3369: done (11 left) 2014-08-08T10:48:51.722 INFO:tasks.rados.rados.0.vpm137.stdout:3372: done (10 left) 2014-08-08T10:48:51.723 INFO:tasks.rados.rados.0.vpm137.stdout:3380: write oid 414 current snap is 0 2014-08-08T10:48:51.723 INFO:tasks.rados.rados.0.vpm137.stdout:3380: seq_num 2091 ranges {560264=615905,1918140=626375,2560263=1} 2014-08-08T10:48:51.747 INFO:tasks.rados.rados.0.vpm137.stdout:3380: writing vpm1375878-414 from 560264 to 1176169 tid 1 2014-08-08T10:48:51.786 INFO:tasks.rados.rados.0.vpm137.stdout:3380: writing vpm1375878-414 from 1918140 to 2544515 tid 2 2014-08-08T10:48:51.788 INFO:tasks.rados.rados.0.vpm137.stdout:3380: writing vpm1375878-414 from 2560263 to 2560264 tid 3 2014-08-08T10:48:51.791 INFO:tasks.rados.rados.0.vpm137.stdout:3374: finishing write tid 1 to vpm1375878-454 2014-08-08T10:48:51.791 INFO:tasks.rados.rados.0.vpm137.stdout:3374: finishing write tid 3 to vpm1375878-454 2014-08-08T10:48:51.791 INFO:tasks.rados.rados.0.vpm137.stdout:3364: finishing write tid 2 to vpm1375878-217 2014-08-08T10:48:51.792 INFO:tasks.rados.rados.0.vpm137.stdout:3364: finishing write tid 3 to vpm1375878-217 2014-08-08T10:48:51.792 INFO:tasks.rados.rados.0.vpm137.stdout:3364: finishing write tid 5 to vpm1375878-217 2014-08-08T10:48:51.792 INFO:tasks.rados.rados.0.vpm137.stdout:3363: finishing write tid 1 to vpm1375878-95 2014-08-08T10:48:51.793 INFO:tasks.rados.rados.0.vpm137.stdout:3374: finishing write tid 4 to vpm1375878-454 2014-08-08T10:48:51.793 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 454 v 468 (ObjNum 2087 snap 0 seq_num 2087) dirty exists 2014-08-08T10:48:51.794 INFO:tasks.rados.rados.0.vpm137.stdout:3363: finishing write tid 2 to vpm1375878-95 2014-08-08T10:48:51.794 INFO:tasks.rados.rados.0.vpm137.stdout:3364: finishing write tid 6 to vpm1375878-217 2014-08-08T10:48:51.794 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 217 v 572 (ObjNum 2082 snap 0 seq_num 2082) dirty exists 2014-08-08T10:48:51.794 INFO:tasks.rados.rados.0.vpm137.stdout:3371: expect (ObjNum 1647 snap 0 seq_num 1647) 2014-08-08T10:48:52.631 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:48:13.452609', u'name': u'a', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89775840, u'health': u'HEALTH_OK', u'kb_used': 9177728, u'store_stats': {u'bytes_total': 23302015, u'bytes_log': 3880155, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 97/1048 objects degraded (9.256%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'recovering_bytes_per_sec': 7913551, u'degraded_objects': 97, u'num_pgs': 258, u'recovering_keys_per_sec': 0, u'data_bytes': 819285954, u'degraded_total': 1048, u'bytes_used': 1615818752, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 886446, u'op_per_sec': 14, u'version': 1654, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 243, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'read_bytes_sec': 1374134, u'degraded_ratio': u'9.256', u'bytes_avail': 427671195648}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 4, u'epoch': 617, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:48:52.632 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:52.904 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:48:52.905 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:53.037 INFO:tasks.rados.rados.0.vpm137.stdout:3364: done (10 left) 2014-08-08T10:48:53.037 INFO:tasks.rados.rados.0.vpm137.stdout:3371: done (9 left) 2014-08-08T10:48:53.037 INFO:tasks.rados.rados.0.vpm137.stdout:3374: done (8 left) 2014-08-08T10:48:53.038 INFO:tasks.rados.rados.0.vpm137.stdout:3381: write oid 340 current snap is 0 2014-08-08T10:48:53.038 INFO:tasks.rados.rados.0.vpm137.stdout:3381: seq_num 2092 ranges {620422=436286,1844753=447994,2891193=723277,3820421=1} 2014-08-08T10:48:53.056 INFO:tasks.rados.rados.0.vpm137.stdout:3381: writing vpm1375878-340 from 620422 to 1056708 tid 1 2014-08-08T10:48:53.078 INFO:tasks.rados.rados.0.vpm137.stdout:3381: writing vpm1375878-340 from 1844753 to 2292747 tid 2 2014-08-08T10:48:53.107 INFO:tasks.rados.rados.0.vpm137.stdout:3381: writing vpm1375878-340 from 2891193 to 3614470 tid 3 2014-08-08T10:48:53.110 INFO:tasks.rados.rados.0.vpm137.stdout:3381: writing vpm1375878-340 from 3820421 to 3820422 tid 4 2014-08-08T10:48:53.112 INFO:tasks.rados.rados.0.vpm137.stdout:3367: finishing write tid 1 to vpm1375878-479 2014-08-08T10:48:53.112 INFO:tasks.rados.rados.0.vpm137.stdout:3367: finishing write tid 2 to vpm1375878-479 2014-08-08T10:48:53.113 INFO:tasks.rados.rados.0.vpm137.stdout:3367: finishing write tid 3 to vpm1375878-479 2014-08-08T10:48:53.113 INFO:tasks.rados.rados.0.vpm137.stdout:3367: finishing write tid 5 to vpm1375878-479 2014-08-08T10:48:53.113 INFO:tasks.rados.rados.0.vpm137.stdout:3363: finishing write tid 4 to vpm1375878-95 2014-08-08T10:48:53.113 INFO:tasks.rados.rados.0.vpm137.stdout:3375: expect (ObjNum 1785 snap 0 seq_num 1785) 2014-08-08T10:48:53.224 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:53.653 INFO:tasks.rados.rados.0.vpm137.stdout:3375: done (8 left) 2014-08-08T10:48:53.653 INFO:tasks.rados.rados.0.vpm137.stdout:3382: read oid 154 snap -1 2014-08-08T10:48:53.653 INFO:tasks.rados.rados.0.vpm137.stdout:3383: read oid 354 snap -1 2014-08-08T10:48:53.654 INFO:tasks.rados.rados.0.vpm137.stdout:3384: write oid 412 current snap is 0 2014-08-08T10:48:53.654 INFO:tasks.rados.rados.0.vpm137.stdout:3384: seq_num 2093 ranges {565250=1} 2014-08-08T10:48:53.658 INFO:tasks.rados.rados.0.vpm137.stdout:3384: writing vpm1375878-412 from 565250 to 565251 tid 1 2014-08-08T10:48:53.659 INFO:tasks.rados.rados.0.vpm137.stdout:3376: finishing write tid 1 to vpm1375878-214 2014-08-08T10:48:53.659 INFO:tasks.rados.rados.0.vpm137.stdout:3376: finishing write tid 2 to vpm1375878-214 2014-08-08T10:48:53.660 INFO:tasks.rados.rados.0.vpm137.stdout:3376: finishing write tid 4 to vpm1375878-214 2014-08-08T10:48:53.660 INFO:tasks.rados.rados.0.vpm137.stdout:3377: finishing write tid 1 to vpm1375878-223 2014-08-08T10:48:53.660 INFO:tasks.rados.rados.0.vpm137.stdout:3377: finishing write tid 2 to vpm1375878-223 2014-08-08T10:48:53.661 INFO:tasks.rados.rados.0.vpm137.stdout:3377: finishing write tid 4 to vpm1375878-223 2014-08-08T10:48:53.664 INFO:tasks.rados.rados.0.vpm137.stdout:3367: finishing write tid 6 to vpm1375878-479 2014-08-08T10:48:53.664 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 479 v 353 (ObjNum 2084 snap 0 seq_num 2084) dirty exists 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:3377: finishing write tid 5 to vpm1375878-223 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 223 v 388 (ObjNum 2089 snap 0 seq_num 2089) dirty exists 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:3376: finishing write tid 5 to vpm1375878-214 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 214 v 356 (ObjNum 2088 snap 0 seq_num 2088) dirty exists 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:3379: finishing write tid 1 to vpm1375878-85 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:3379: finishing write tid 2 to vpm1375878-85 2014-08-08T10:48:53.667 INFO:tasks.rados.rados.0.vpm137.stdout:3379: finishing write tid 3 to vpm1375878-85 2014-08-08T10:48:53.668 INFO:tasks.rados.rados.0.vpm137.stdout:3379: finishing write tid 5 to vpm1375878-85 2014-08-08T10:48:53.668 INFO:tasks.rados.rados.0.vpm137.stdout:3363: finishing write tid 5 to vpm1375878-95 2014-08-08T10:48:53.668 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 95 v 575 (ObjNum 2081 snap 0 seq_num 2081) dirty exists 2014-08-08T10:48:53.668 INFO:tasks.rados.rados.0.vpm137.stdout:3379: finishing write tid 6 to vpm1375878-85 2014-08-08T10:48:53.668 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 85 v 384 (ObjNum 2090 snap 0 seq_num 2090) dirty exists 2014-08-08T10:48:53.669 INFO:tasks.rados.rados.0.vpm137.stdout:3378: expect (ObjNum 830 snap 0 seq_num 830) 2014-08-08T10:48:55.809 INFO:tasks.rados.rados.0.vpm137.stdout:3363: done (10 left) 2014-08-08T10:48:55.810 INFO:tasks.rados.rados.0.vpm137.stdout:3367: done (9 left) 2014-08-08T10:48:55.810 INFO:tasks.rados.rados.0.vpm137.stdout:3376: done (8 left) 2014-08-08T10:48:55.811 INFO:tasks.rados.rados.0.vpm137.stdout:3377: done (7 left) 2014-08-08T10:48:55.811 INFO:tasks.rados.rados.0.vpm137.stdout:3378: done (6 left) 2014-08-08T10:48:55.811 INFO:tasks.rados.rados.0.vpm137.stdout:3379: done (5 left) 2014-08-08T10:48:55.811 INFO:tasks.rados.rados.0.vpm137.stdout:3385: read oid 179 snap -1 2014-08-08T10:48:55.812 INFO:tasks.rados.rados.0.vpm137.stdout:3386: write oid 140 current snap is 0 2014-08-08T10:48:55.812 INFO:tasks.rados.rados.0.vpm137.stdout:3386: seq_num 2094 ranges {261504=1} 2014-08-08T10:48:55.816 INFO:tasks.rados.rados.0.vpm137.stdout:3386: writing vpm1375878-140 from 261504 to 261505 tid 1 2014-08-08T10:48:55.817 INFO:tasks.rados.rados.0.vpm137.stdout:3380: finishing write tid 1 to vpm1375878-414 2014-08-08T10:48:55.817 INFO:tasks.rados.rados.0.vpm137.stdout:3380: finishing write tid 2 to vpm1375878-414 2014-08-08T10:48:55.817 INFO:tasks.rados.rados.0.vpm137.stdout:3380: finishing write tid 3 to vpm1375878-414 2014-08-08T10:48:55.817 INFO:tasks.rados.rados.0.vpm137.stdout:3380: finishing write tid 5 to vpm1375878-414 2014-08-08T10:48:55.818 INFO:tasks.rados.rados.0.vpm137.stdout:3380: finishing write tid 6 to vpm1375878-414 2014-08-08T10:48:55.818 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 414 v 472 (ObjNum 2091 snap 0 seq_num 2091) dirty exists 2014-08-08T10:48:55.819 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 1 to vpm1375878-340 2014-08-08T10:48:55.819 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 2 to vpm1375878-340 2014-08-08T10:48:55.819 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 3 to vpm1375878-340 2014-08-08T10:48:55.819 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 4 to vpm1375878-340 2014-08-08T10:48:55.819 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 6 to vpm1375878-340 2014-08-08T10:48:55.820 INFO:tasks.rados.rados.0.vpm137.stdout:3381: finishing write tid 7 to vpm1375878-340 2014-08-08T10:48:55.820 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 340 v 303 (ObjNum 2092 snap 0 seq_num 2092) dirty exists 2014-08-08T10:48:55.820 INFO:tasks.rados.rados.0.vpm137.stdout:3383: expect (ObjNum 1426 snap 0 seq_num 1426) 2014-08-08T10:48:56.251 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:48:57.318 INFO:tasks.rados.rados.0.vpm137.stdout:3380: done (6 left) 2014-08-08T10:48:57.318 INFO:tasks.rados.rados.0.vpm137.stdout:3381: done (5 left) 2014-08-08T10:48:57.319 INFO:tasks.rados.rados.0.vpm137.stdout:3383: done (4 left) 2014-08-08T10:48:57.319 INFO:tasks.rados.rados.0.vpm137.stdout:3387: read oid 17 snap -1 2014-08-08T10:48:57.319 INFO:tasks.rados.rados.0.vpm137.stdout:3388: read oid 332 snap -1 2014-08-08T10:48:57.319 INFO:tasks.rados.rados.0.vpm137.stdout:3389: write oid 340 current snap is 0 2014-08-08T10:48:57.320 INFO:tasks.rados.rados.0.vpm137.stdout:3389: seq_num 2095 ranges {756735=539219,1902477=657556,3093846=762401,3956734=1} 2014-08-08T10:48:57.357 INFO:tasks.rados.rados.0.vpm137.stdout:3389: writing vpm1375878-340 from 756735 to 1295954 tid 1 2014-08-08T10:48:57.401 INFO:tasks.rados.rados.0.vpm137.stdout:3389: writing vpm1375878-340 from 1902477 to 2560033 tid 2 2014-08-08T10:48:57.434 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:48:57.451 INFO:tasks.rados.rados.0.vpm137.stdout:3389: writing vpm1375878-340 from 3093846 to 3856247 tid 3 2014-08-08T10:48:57.453 INFO:tasks.rados.rados.0.vpm137.stdout:3389: writing vpm1375878-340 from 3956734 to 3956735 tid 4 2014-08-08T10:48:57.455 INFO:tasks.rados.rados.0.vpm137.stdout:3382: expect (ObjNum 1400 snap 0 seq_num 1400) 2014-08-08T10:48:57.553 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:48:57.688 INFO:tasks.rados.rados.0.vpm137.stdout:3382: done (6 left) 2014-08-08T10:48:57.688 INFO:tasks.rados.rados.0.vpm137.stdout:3390: read oid 290 snap -1 2014-08-08T10:48:57.689 INFO:tasks.rados.rados.0.vpm137.stdout:3391: read oid 112 snap -1 2014-08-08T10:48:57.689 INFO:tasks.rados.rados.0.vpm137.stdout:3392: read oid 363 snap -1 2014-08-08T10:48:57.689 INFO:tasks.rados.rados.0.vpm137.stdout:3393: write oid 224 current snap is 0 2014-08-08T10:48:57.690 INFO:tasks.rados.rados.0.vpm137.stdout:3393: seq_num 2096 ranges {549117=512047,1562337=186780} 2014-08-08T10:48:57.714 INFO:tasks.rados.rados.0.vpm137.stdout:3393: writing vpm1375878-224 from 549117 to 1061164 tid 1 2014-08-08T10:48:57.728 INFO:tasks.rados.rados.0.vpm137.stdout:3393: writing vpm1375878-224 from 1562337 to 1749117 tid 2 2014-08-08T10:48:57.732 INFO:tasks.rados.rados.0.vpm137.stdout:3384: finishing write tid 1 to vpm1375878-412 2014-08-08T10:48:57.732 INFO:tasks.rados.rados.0.vpm137.stdout:3384: finishing write tid 3 to vpm1375878-412 2014-08-08T10:48:57.733 INFO:tasks.rados.rados.0.vpm137.stdout:3384: finishing write tid 4 to vpm1375878-412 2014-08-08T10:48:57.733 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 412 v 386 (ObjNum 2093 snap 0 seq_num 2093) dirty exists 2014-08-08T10:48:57.733 INFO:tasks.rados.rados.0.vpm137.stdout:3386: finishing write tid 1 to vpm1375878-140 2014-08-08T10:48:57.733 INFO:tasks.rados.rados.0.vpm137.stdout:3386: finishing write tid 3 to vpm1375878-140 2014-08-08T10:48:57.733 INFO:tasks.rados.rados.0.vpm137.stdout:3386: finishing write tid 4 to vpm1375878-140 2014-08-08T10:48:57.734 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 140 v 328 (ObjNum 2094 snap 0 seq_num 2094) dirty exists 2014-08-08T10:48:57.734 INFO:tasks.rados.rados.0.vpm137.stdout:3387: expect (ObjNum 2046 snap 0 seq_num 2046) 2014-08-08T10:48:57.804 INFO:tasks.rados.rados.0.vpm137.stdout:3384: done (9 left) 2014-08-08T10:48:57.804 INFO:tasks.rados.rados.0.vpm137.stdout:3385: done (8 left) 2014-08-08T10:48:57.805 INFO:tasks.rados.rados.0.vpm137.stdout:3386: done (7 left) 2014-08-08T10:48:57.805 INFO:tasks.rados.rados.0.vpm137.stdout:3387: done (6 left) 2014-08-08T10:48:57.805 INFO:tasks.rados.rados.0.vpm137.stdout:3394: delete oid 396 current snap is 0 2014-08-08T10:48:57.805 INFO:tasks.rados.rados.0.vpm137.stdout:3388: expect (ObjNum 1438 snap 0 seq_num 1438) 2014-08-08T10:48:59.586 INFO:tasks.rados.rados.0.vpm137.stdout:3388: done (6 left) 2014-08-08T10:48:59.586 INFO:tasks.rados.rados.0.vpm137.stdout:3394: done (5 left) 2014-08-08T10:48:59.586 INFO:tasks.rados.rados.0.vpm137.stdout:3395: read oid 492 snap -1 2014-08-08T10:48:59.587 INFO:tasks.rados.rados.0.vpm137.stdout:3396: read oid 20 snap -1 2014-08-08T10:48:59.587 INFO:tasks.rados.rados.0.vpm137.stdout:3397: read oid 249 snap -1 2014-08-08T10:48:59.587 INFO:tasks.rados.rados.0.vpm137.stdout:3398: write oid 205 current snap is 0 2014-08-08T10:48:59.587 INFO:tasks.rados.rados.0.vpm137.stdout:3398: seq_num 2098 ranges {456952=490728,1656951=1} 2014-08-08T10:48:59.612 INFO:tasks.rados.rados.0.vpm137.stdout:3398: writing vpm1375878-205 from 456952 to 947680 tid 1 2014-08-08T10:48:59.638 INFO:tasks.rados.rados.0.vpm137.stdout:3398: writing vpm1375878-205 from 1656951 to 1656952 tid 2 2014-08-08T10:48:59.641 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 1 to vpm1375878-340 2014-08-08T10:48:59.641 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 2 to vpm1375878-340 2014-08-08T10:48:59.641 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 3 to vpm1375878-340 2014-08-08T10:48:59.641 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 4 to vpm1375878-340 2014-08-08T10:48:59.642 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 6 to vpm1375878-340 2014-08-08T10:48:59.642 INFO:tasks.rados.rados.0.vpm137.stdout:3389: finishing write tid 7 to vpm1375878-340 2014-08-08T10:48:59.642 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 340 v 308 (ObjNum 2095 snap 0 seq_num 2095) dirty exists 2014-08-08T10:48:59.642 INFO:tasks.rados.rados.0.vpm137.stdout:3390: expect (ObjNum 1652 snap 0 seq_num 1652) 2014-08-08T10:48:59.687 INFO:tasks.rados.rados.0.vpm137.stdout:3389: done (8 left) 2014-08-08T10:48:59.687 INFO:tasks.rados.rados.0.vpm137.stdout:3390: done (7 left) 2014-08-08T10:48:59.687 INFO:tasks.rados.rados.0.vpm137.stdout:3399: read oid 159 snap -1 2014-08-08T10:48:59.687 INFO:tasks.rados.rados.0.vpm137.stdout:3400: write oid 103 current snap is 0 2014-08-08T10:48:59.688 INFO:tasks.rados.rados.0.vpm137.stdout:3400: seq_num 2099 ranges {650358=613956,1868638=724832,3328123=122235} 2014-08-08T10:48:59.721 INFO:tasks.rados.rados.0.vpm137.stdout:3400: writing vpm1375878-103 from 650358 to 1264314 tid 1 2014-08-08T10:48:59.762 INFO:tasks.rados.rados.0.vpm137.stdout:3400: writing vpm1375878-103 from 1868638 to 2593470 tid 2 2014-08-08T10:48:59.777 INFO:tasks.rados.rados.0.vpm137.stdout:3400: writing vpm1375878-103 from 3328123 to 3450358 tid 3 2014-08-08T10:48:59.779 INFO:tasks.rados.rados.0.vpm137.stdout:3392: expect (ObjNum 1745 snap 0 seq_num 1745) 2014-08-08T10:49:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:3392: done (8 left) 2014-08-08T10:49:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:3401: read oid 86 snap -1 2014-08-08T10:49:00.523 INFO:tasks.rados.rados.0.vpm137.stdout:3402: read oid 450 snap -1 2014-08-08T10:49:00.523 INFO:tasks.rados.rados.0.vpm137.stdout:3403: read oid 489 snap -1 2014-08-08T10:49:00.523 INFO:tasks.rados.rados.0.vpm137.stdout:3404: read oid 307 snap -1 2014-08-08T10:49:00.524 INFO:tasks.rados.rados.0.vpm137.stdout:3405: read oid 332 snap -1 2014-08-08T10:49:00.524 INFO:tasks.rados.rados.0.vpm137.stdout:3406: delete oid 494 current snap is 0 2014-08-08T10:49:00.525 INFO:tasks.rados.rados.0.vpm137.stdout:3391: expect (ObjNum 1753 snap 0 seq_num 1753) 2014-08-08T10:49:00.890 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:48:13.452609', u'name': u'a', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89775840, u'health': u'HEALTH_OK', u'kb_used': 9177728, u'store_stats': {u'bytes_total': 23302015, u'bytes_log': 3880155, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'recovering_bytes_per_sec': 6822209, u'num_pgs': 258, u'recovering_keys_per_sec': 0, u'data_bytes': 824207606, u'bytes_used': 1604501504, u'recovering_objects_per_sec': 2, u'op_per_sec': 15, u'version': 1656, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2861915, u'write_bytes_sec': 1011278, u'bytes_avail': 427682512896}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 4, u'epoch': 617, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:00.891 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:01.099 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:49:01.099 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:01.301 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:01.688 INFO:tasks.rados.rados.0.vpm137.stdout:3391: done (13 left) 2014-08-08T10:49:01.688 INFO:tasks.rados.rados.0.vpm137.stdout:3406: done (12 left) 2014-08-08T10:49:01.689 INFO:tasks.rados.rados.0.vpm137.stdout:3407: read oid 218 snap -1 2014-08-08T10:49:01.689 INFO:tasks.rados.rados.0.vpm137.stdout:3408: write oid 336 current snap is 0 2014-08-08T10:49:01.689 INFO:tasks.rados.rados.0.vpm137.stdout:3408: seq_num 2101 ranges {558193=710013,1681030=465223,2799374=558819} 2014-08-08T10:49:01.728 INFO:tasks.rados.rados.0.vpm137.stdout:3408: writing vpm1375878-336 from 558193 to 1268206 tid 1 2014-08-08T10:49:01.754 INFO:tasks.rados.rados.0.vpm137.stdout:3408: writing vpm1375878-336 from 1681030 to 2146253 tid 2 2014-08-08T10:49:01.778 INFO:tasks.rados.rados.0.vpm137.stdout:3408: writing vpm1375878-336 from 2799374 to 3358193 tid 3 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:3393: finishing write tid 1 to vpm1375878-224 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:3393: finishing write tid 2 to vpm1375878-224 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:3393: finishing write tid 4 to vpm1375878-224 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:3393: finishing write tid 5 to vpm1375878-224 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 224 v 380 (ObjNum 2096 snap 0 seq_num 2096) dirty exists 2014-08-08T10:49:01.782 INFO:tasks.rados.rados.0.vpm137.stdout:3396: expect (ObjNum 1726 snap 0 seq_num 1726) 2014-08-08T10:49:03.446 INFO:tasks.rados.rados.0.vpm137.stdout:3393: done (13 left) 2014-08-08T10:49:03.447 INFO:tasks.rados.rados.0.vpm137.stdout:3396: done (12 left) 2014-08-08T10:49:03.447 INFO:tasks.rados.rados.0.vpm137.stdout:3409: read oid 324 snap -1 2014-08-08T10:49:03.447 INFO:tasks.rados.rados.0.vpm137.stdout:3410: write oid 460 current snap is 0 2014-08-08T10:49:03.447 INFO:tasks.rados.rados.0.vpm137.stdout:3410: seq_num 2102 ranges {750575=400000} 2014-08-08T10:49:03.466 INFO:tasks.rados.rados.0.vpm137.stdout:3410: writing vpm1375878-460 from 750575 to 1150575 tid 1 2014-08-08T10:49:03.471 INFO:tasks.rados.rados.0.vpm137.stdout:3397: expect (ObjNum 1603 snap 0 seq_num 1603) 2014-08-08T10:49:04.333 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:04.560 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:04.606 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:04.882 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:48:13.452609', u'name': u'a', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89775840, u'health': u'HEALTH_OK', u'kb_used': 9177728, u'store_stats': {u'bytes_total': 23302015, u'bytes_log': 3880155, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'recovering_bytes_per_sec': 6822209, u'num_pgs': 258, u'recovering_keys_per_sec': 0, u'data_bytes': 824207606, u'bytes_used': 1604501504, u'recovering_objects_per_sec': 2, u'op_per_sec': 15, u'version': 1656, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2861915, u'write_bytes_sec': 1011278, u'bytes_avail': 427682512896}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 4, u'epoch': 617, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:04.883 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:05.732 INFO:tasks.rados.rados.0.vpm137.stdout:3397: done (13 left) 2014-08-08T10:49:05.732 INFO:tasks.rados.rados.0.vpm137.stdout:3411: read oid 424 snap -1 2014-08-08T10:49:05.732 INFO:tasks.rados.rados.0.vpm137.stdout:3412: write oid 491 current snap is 0 2014-08-08T10:49:05.733 INFO:tasks.rados.rados.0.vpm137.stdout:3412: seq_num 2103 ranges {446829=400000} 2014-08-08T10:49:05.750 INFO:tasks.rados.rados.0.vpm137.stdout:3412: writing vpm1375878-491 from 446829 to 846829 tid 1 2014-08-08T10:49:05.752 INFO:tasks.rados.rados.0.vpm137.stdout:3399: expect (ObjNum 1912 snap 0 seq_num 1912) 2014-08-08T10:49:07.395 INFO:tasks.rados.rados.0.vpm137.stdout:3399: done (14 left) 2014-08-08T10:49:07.395 INFO:tasks.rados.rados.0.vpm137.stdout:3413: read oid 257 snap -1 2014-08-08T10:49:07.395 INFO:tasks.rados.rados.0.vpm137.stdout:3414: read oid 400 snap -1 2014-08-08T10:49:07.396 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:07.397 INFO:tasks.rados.rados.0.vpm137.stdout:3398: finishing write tid 1 to vpm1375878-205 2014-08-08T10:49:07.397 INFO:tasks.rados.rados.0.vpm137.stdout:3395: expect (ObjNum 1200 snap 0 seq_num 1200) 2014-08-08T10:49:08.111 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T10:49:08.111 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:08.344 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:09.483 INFO:tasks.rados.rados.0.vpm137.stdout:3395: done (15 left) 2014-08-08T10:49:09.484 INFO:tasks.rados.rados.0.vpm137.stdout:3415: read oid 91 snap -1 2014-08-08T10:49:09.484 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:09.485 INFO:tasks.rados.rados.0.vpm137.stdout:3398: finishing write tid 2 to vpm1375878-205 2014-08-08T10:49:09.486 INFO:tasks.rados.rados.0.vpm137.stdout:3398: finishing write tid 4 to vpm1375878-205 2014-08-08T10:49:09.486 INFO:tasks.rados.rados.0.vpm137.stdout:3400: finishing write tid 1 to vpm1375878-103 2014-08-08T10:49:09.486 INFO:tasks.rados.rados.0.vpm137.stdout:3398: finishing write tid 5 to vpm1375878-205 2014-08-08T10:49:09.486 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 205 v 391 (ObjNum 2098 snap 0 seq_num 2098) dirty exists 2014-08-08T10:49:09.487 INFO:tasks.rados.rados.0.vpm137.stdout:3398: done (15 left) 2014-08-08T10:49:09.488 INFO:tasks.rados.rados.0.vpm137.stdout:3416: read oid 266 snap -1 2014-08-08T10:49:09.488 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:09.488 INFO:tasks.rados.rados.0.vpm137.stdout:3400: finishing write tid 2 to vpm1375878-103 2014-08-08T10:49:09.488 INFO:tasks.rados.rados.0.vpm137.stdout:3400: finishing write tid 3 to vpm1375878-103 2014-08-08T10:49:09.488 INFO:tasks.rados.rados.0.vpm137.stdout:3400: finishing write tid 5 to vpm1375878-103 2014-08-08T10:49:09.489 INFO:tasks.rados.rados.0.vpm137.stdout:3400: finishing write tid 6 to vpm1375878-103 2014-08-08T10:49:09.489 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 103 v 476 (ObjNum 2099 snap 0 seq_num 2099) dirty exists 2014-08-08T10:49:09.490 INFO:tasks.rados.rados.0.vpm137.stdout:3400: done (15 left) 2014-08-08T10:49:09.490 INFO:tasks.rados.rados.0.vpm137.stdout:3417: write oid 304 current snap is 0 2014-08-08T10:49:09.490 INFO:tasks.rados.rados.0.vpm137.stdout:3417: seq_num 2104 ranges {542058=1} 2014-08-08T10:49:09.503 INFO:tasks.rados.rados.0.vpm137.stdout:3417: writing vpm1375878-304 from 542058 to 542059 tid 1 2014-08-08T10:49:09.505 INFO:tasks.rados.rados.0.vpm137.stdout:3404: expect (ObjNum 616 snap 0 seq_num 616) 2014-08-08T10:49:09.799 INFO:tasks.rados.rados.0.vpm137.stdout:3402: done (15 left) 2014-08-08T10:49:09.799 INFO:tasks.rados.rados.0.vpm137.stdout:3404: done (14 left) 2014-08-08T10:49:09.800 INFO:tasks.rados.rados.0.vpm137.stdout:3418: read oid 140 snap -1 2014-08-08T10:49:09.800 INFO:tasks.rados.rados.0.vpm137.stdout:3419: write oid 74 current snap is 0 2014-08-08T10:49:09.800 INFO:tasks.rados.rados.0.vpm137.stdout:3419: seq_num 2105 ranges {734440=633326,1893718=440722} 2014-08-08T10:49:09.828 INFO:tasks.rados.rados.0.vpm137.stdout:3419: writing vpm1375878-74 from 734440 to 1367766 tid 1 2014-08-08T10:49:09.853 INFO:tasks.rados.rados.0.vpm137.stdout:3419: writing vpm1375878-74 from 1893718 to 2334440 tid 2 2014-08-08T10:49:09.855 INFO:tasks.rados.rados.0.vpm137.stdout:3405: expect (ObjNum 1438 snap 0 seq_num 1438) 2014-08-08T10:49:11.372 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:11.675 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:11.718 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:11.948 INFO:tasks.rados.rados.0.vpm137.stdout:3405: done (15 left) 2014-08-08T10:49:11.948 INFO:tasks.rados.rados.0.vpm137.stdout:3420: write oid 334 current snap is 0 2014-08-08T10:49:11.949 INFO:tasks.rados.rados.0.vpm137.stdout:3420: seq_num 2106 ranges {547046=488778,1607850=719047,2547045=1} 2014-08-08T10:49:11.981 INFO:tasks.rados.rados.0.vpm137.stdout:3420: writing vpm1375878-334 from 547046 to 1035824 tid 1 2014-08-08T10:49:12.029 INFO:tasks.rados.rados.0.vpm137.stdout:3420: writing vpm1375878-334 from 1607850 to 2326897 tid 2 2014-08-08T10:49:12.034 INFO:tasks.rados.rados.0.vpm137.stdout:3420: writing vpm1375878-334 from 2547045 to 2547046 tid 3 2014-08-08T10:49:12.035 INFO:tasks.rados.rados.0.vpm137.stdout:3401: expect (ObjNum 1408 snap 0 seq_num 1408) 2014-08-08T10:49:12.437 INFO:tasks.rados.rados.0.vpm137.stdout:3401: done (15 left) 2014-08-08T10:49:12.438 INFO:tasks.rados.rados.0.vpm137.stdout:3421: write oid 48 current snap is 0 2014-08-08T10:49:12.438 INFO:tasks.rados.rados.0.vpm137.stdout:3421: seq_num 2107 ranges {642276=613031,1873407=368869} 2014-08-08T10:49:12.477 INFO:tasks.rados.rados.0.vpm137.stdout:3421: writing vpm1375878-48 from 642276 to 1255307 tid 1 2014-08-08T10:49:12.507 INFO:tasks.rados.rados.0.vpm137.stdout:3421: writing vpm1375878-48 from 1873407 to 2242276 tid 2 2014-08-08T10:49:12.511 INFO:tasks.rados.rados.0.vpm137.stdout:3403: expect (ObjNum 993 snap 0 seq_num 993) 2014-08-08T10:49:13.375 INFO:tasks.rados.rados.0.vpm137.stdout:3403: done (15 left) 2014-08-08T10:49:13.376 INFO:tasks.rados.rados.0.vpm137.stdout:3422: write oid 439 current snap is 0 2014-08-08T10:49:13.376 INFO:tasks.rados.rados.0.vpm137.stdout:3422: seq_num 2108 ranges {587106=584835,1801043=494509,2934479=52627} 2014-08-08T10:49:13.415 INFO:tasks.rados.rados.0.vpm137.stdout:3422: writing vpm1375878-439 from 587106 to 1171941 tid 1 2014-08-08T10:49:13.443 INFO:tasks.rados.rados.0.vpm137.stdout:3422: writing vpm1375878-439 from 1801043 to 2295552 tid 2 2014-08-08T10:49:13.451 INFO:tasks.rados.rados.0.vpm137.stdout:3422: writing vpm1375878-439 from 2934479 to 2987106 tid 3 2014-08-08T10:49:13.453 INFO:tasks.rados.rados.0.vpm137.stdout:3408: finishing write tid 1 to vpm1375878-336 2014-08-08T10:49:13.453 INFO:tasks.rados.rados.0.vpm137.stdout:3408: finishing write tid 2 to vpm1375878-336 2014-08-08T10:49:13.453 INFO:tasks.rados.rados.0.vpm137.stdout:3408: finishing write tid 3 to vpm1375878-336 2014-08-08T10:49:13.454 INFO:tasks.rados.rados.0.vpm137.stdout:3407: expect (ObjNum 1885 snap 0 seq_num 1885) 2014-08-08T10:49:14.921 INFO:tasks.rados.rados.0.vpm137.stdout:3407: done (15 left) 2014-08-08T10:49:14.922 INFO:tasks.rados.rados.0.vpm137.stdout:3423: read oid 306 snap -1 2014-08-08T10:49:14.922 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:14.923 INFO:tasks.rados.rados.0.vpm137.stdout:3408: finishing write tid 5 to vpm1375878-336 2014-08-08T10:49:14.923 INFO:tasks.rados.rados.0.vpm137.stdout:3408: finishing write tid 6 to vpm1375878-336 2014-08-08T10:49:14.923 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 336 v 292 (ObjNum 2101 snap 0 seq_num 2101) dirty exists 2014-08-08T10:49:14.924 INFO:tasks.rados.rados.0.vpm137.stdout:3408: done (15 left) 2014-08-08T10:49:14.924 INFO:tasks.rados.rados.0.vpm137.stdout:3424: read oid 342 snap -1 2014-08-08T10:49:14.924 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:14.925 INFO:tasks.rados.rados.0.vpm137.stdout:3409: done (15 left) 2014-08-08T10:49:14.925 INFO:tasks.rados.rados.0.vpm137.stdout:3425: write oid 345 current snap is 0 2014-08-08T10:49:14.925 INFO:tasks.rados.rados.0.vpm137.stdout:3425: seq_num 2109 ranges {531935=708063,1763750=564704,2735703=487036,3731934=1} 2014-08-08T10:49:14.967 INFO:tasks.rados.rados.0.vpm137.stdout:3425: writing vpm1375878-345 from 531935 to 1239998 tid 1 2014-08-08T10:49:14.969 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:48:17.561188', u'name': u'b', u'avail_percent': 87, u'kb_total': 103178528, u'kb_avail': 89774456, u'health': u'HEALTH_OK', u'kb_used': 9179112, u'store_stats': {u'bytes_total': 25314095, u'bytes_log': 5892291, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 826405260, u'bytes_used': 1786134528, u'op_per_sec': 10, u'version': 1660, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1390814, u'write_bytes_sec': 327758, u'bytes_avail': 427500879872}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:14.970 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:15.001 INFO:tasks.rados.rados.0.vpm137.stdout:3425: writing vpm1375878-345 from 1763750 to 2328454 tid 2 2014-08-08T10:49:15.025 INFO:tasks.rados.rados.0.vpm137.stdout:3425: writing vpm1375878-345 from 2735703 to 3222739 tid 3 2014-08-08T10:49:15.033 INFO:tasks.rados.rados.0.vpm137.stdout:3425: writing vpm1375878-345 from 3731934 to 3731935 tid 4 2014-08-08T10:49:15.035 INFO:tasks.rados.rados.0.vpm137.stdout:3410: finishing write tid 1 to vpm1375878-460 2014-08-08T10:49:15.035 INFO:tasks.rados.rados.0.vpm137.stdout:3410: finishing write tid 3 to vpm1375878-460 2014-08-08T10:49:15.035 INFO:tasks.rados.rados.0.vpm137.stdout:3410: finishing write tid 4 to vpm1375878-460 2014-08-08T10:49:15.035 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 460 v 331 (ObjNum 2102 snap 0 seq_num 2102) dirty exists 2014-08-08T10:49:15.035 INFO:tasks.rados.rados.0.vpm137.stdout:3411: expect (ObjNum 2053 snap 0 seq_num 2053) 2014-08-08T10:49:15.229 INFO:tasks.rados.rados.0.vpm137.stdout:3410: done (15 left) 2014-08-08T10:49:15.229 INFO:tasks.rados.rados.0.vpm137.stdout:3411: done (14 left) 2014-08-08T10:49:15.229 INFO:tasks.rados.rados.0.vpm137.stdout:3426: delete oid 277 current snap is 0 2014-08-08T10:49:15.230 INFO:tasks.rados.rados.0.vpm137.stdout:3412: finishing write tid 1 to vpm1375878-491 2014-08-08T10:49:15.230 INFO:tasks.rados.rados.0.vpm137.stdout:3412: finishing write tid 3 to vpm1375878-491 2014-08-08T10:49:15.230 INFO:tasks.rados.rados.0.vpm137.stdout:3412: finishing write tid 4 to vpm1375878-491 2014-08-08T10:49:15.230 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 491 v 478 (ObjNum 2103 snap 0 seq_num 2103) dirty exists 2014-08-08T10:49:15.230 INFO:tasks.rados.rados.0.vpm137.stdout:3413: expect (ObjNum 1702 snap 0 seq_num 1702) 2014-08-08T10:49:15.255 INFO:tasks.rados.rados.0.vpm137.stdout:3414: expect (ObjNum 1519 snap 0 seq_num 1519) 2014-08-08T10:49:16.653 INFO:tasks.rados.rados.0.vpm137.stdout:3412: done (14 left) 2014-08-08T10:49:16.654 INFO:tasks.rados.rados.0.vpm137.stdout:3413: done (13 left) 2014-08-08T10:49:16.654 INFO:tasks.rados.rados.0.vpm137.stdout:3414: done (12 left) 2014-08-08T10:49:16.654 INFO:tasks.rados.rados.0.vpm137.stdout:3426: done (11 left) 2014-08-08T10:49:16.654 INFO:tasks.rados.rados.0.vpm137.stdout:3427: delete oid 122 current snap is 0 2014-08-08T10:49:16.654 INFO:tasks.rados.rados.0.vpm137.stdout:3416: expect (ObjNum 1849 snap 0 seq_num 1849) 2014-08-08T10:49:16.673 INFO:tasks.rados.rados.0.vpm137.stdout:3415: expect (ObjNum 1933 snap 0 seq_num 1933) 2014-08-08T10:49:17.625 INFO:tasks.rados.rados.0.vpm137.stdout:3417: finishing write tid 1 to vpm1375878-304 2014-08-08T10:49:17.626 INFO:tasks.rados.rados.0.vpm137.stdout:3417: finishing write tid 3 to vpm1375878-304 2014-08-08T10:49:17.626 INFO:tasks.rados.rados.0.vpm137.stdout:3417: finishing write tid 4 to vpm1375878-304 2014-08-08T10:49:17.626 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 304 v 401 (ObjNum 2104 snap 0 seq_num 2104) dirty exists 2014-08-08T10:49:17.626 INFO:tasks.rados.rados.0.vpm137.stdout:3418: expect (ObjNum 2094 snap 0 seq_num 2094) 2014-08-08T10:49:17.648 INFO:tasks.rados.rados.0.vpm137.stdout:3419: finishing write tid 1 to vpm1375878-74 2014-08-08T10:49:17.648 INFO:tasks.rados.rados.0.vpm137.stdout:3419: finishing write tid 2 to vpm1375878-74 2014-08-08T10:49:17.648 INFO:tasks.rados.rados.0.vpm137.stdout:3419: finishing write tid 4 to vpm1375878-74 2014-08-08T10:49:17.649 INFO:tasks.rados.rados.0.vpm137.stdout:3419: finishing write tid 5 to vpm1375878-74 2014-08-08T10:49:17.649 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 74 v 417 (ObjNum 2105 snap 0 seq_num 2105) dirty exists 2014-08-08T10:49:17.650 INFO:tasks.rados.rados.0.vpm137.stdout:3420: finishing write tid 1 to vpm1375878-334 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3420: finishing write tid 2 to vpm1375878-334 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3420: finishing write tid 3 to vpm1375878-334 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3420: finishing write tid 5 to vpm1375878-334 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3421: finishing write tid 1 to vpm1375878-48 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3421: finishing write tid 2 to vpm1375878-48 2014-08-08T10:49:17.651 INFO:tasks.rados.rados.0.vpm137.stdout:3421: finishing write tid 4 to vpm1375878-48 2014-08-08T10:49:17.652 INFO:tasks.rados.rados.0.vpm137.stdout:3420: finishing write tid 6 to vpm1375878-334 2014-08-08T10:49:17.652 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 334 v 579 (ObjNum 2106 snap 0 seq_num 2106) dirty exists 2014-08-08T10:49:17.652 INFO:tasks.rados.rados.0.vpm137.stdout:3422: finishing write tid 1 to vpm1375878-439 2014-08-08T10:49:17.652 INFO:tasks.rados.rados.0.vpm137.stdout:3422: finishing write tid 2 to vpm1375878-439 2014-08-08T10:49:17.652 INFO:tasks.rados.rados.0.vpm137.stdout:3422: finishing write tid 3 to vpm1375878-439 2014-08-08T10:49:17.653 INFO:tasks.rados.rados.0.vpm137.stdout:3422: finishing write tid 5 to vpm1375878-439 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 1 to vpm1375878-345 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 2 to vpm1375878-345 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 3 to vpm1375878-345 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 4 to vpm1375878-345 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 6 to vpm1375878-345 2014-08-08T10:49:17.654 INFO:tasks.rados.rados.0.vpm137.stdout:3421: finishing write tid 5 to vpm1375878-48 2014-08-08T10:49:17.655 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 48 v 394 (ObjNum 2107 snap 0 seq_num 2107) dirty exists 2014-08-08T10:49:17.655 INFO:tasks.rados.rados.0.vpm137.stdout:3422: finishing write tid 6 to vpm1375878-439 2014-08-08T10:49:17.655 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 439 v 360 (ObjNum 2108 snap 0 seq_num 2108) dirty exists 2014-08-08T10:49:17.655 INFO:tasks.rados.rados.0.vpm137.stdout:3424: expect (ObjNum 1746 snap 0 seq_num 1746) 2014-08-08T10:49:18.204 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:18.205 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:19.018 INFO:tasks.rados.rados.0.vpm137.stdout:3415: done (11 left) 2014-08-08T10:49:19.652 INFO:tasks.rados.rados.0.vpm137.stdout:3416: done (10 left) 2014-08-08T10:49:19.652 INFO:tasks.rados.rados.0.vpm137.stdout:3417: done (9 left) 2014-08-08T10:49:19.655 INFO:tasks.rados.rados.0.vpm137.stdout:3418: done (8 left) 2014-08-08T10:49:19.655 INFO:tasks.rados.rados.0.vpm137.stdout:3419: done (7 left) 2014-08-08T10:49:19.655 INFO:tasks.rados.rados.0.vpm137.stdout:3420: done (6 left) 2014-08-08T10:49:19.655 INFO:tasks.rados.rados.0.vpm137.stdout:3421: done (5 left) 2014-08-08T10:49:19.655 INFO:tasks.rados.rados.0.vpm137.stdout:3422: done (4 left) 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3424: done (3 left) 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3427: done (2 left) 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3428: write oid 258 current snap is 0 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3428: seq_num 2112 ranges {631128=660596,1431127=1} 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3428: writing vpm1375878-258 from 631128 to 1291724 tid 1 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3428: writing vpm1375878-258 from 1431127 to 1431128 tid 2 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3423: expect (ObjNum 1719 snap 0 seq_num 1719) 2014-08-08T10:49:19.656 INFO:tasks.rados.rados.0.vpm137.stdout:3423: done (2 left) 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3429: read oid 11 snap -1 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3430: write oid 311 current snap is 0 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3430: seq_num 2113 ranges {575958=632401,1996663=179295} 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3430: writing vpm1375878-311 from 575958 to 1208359 tid 1 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3430: writing vpm1375878-311 from 1996663 to 2175958 tid 2 2014-08-08T10:49:19.657 INFO:tasks.rados.rados.0.vpm137.stdout:3425: finishing write tid 7 to vpm1375878-345 2014-08-08T10:49:19.658 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 345 v 365 (ObjNum 2109 snap 0 seq_num 2109) dirty exists 2014-08-08T10:49:19.658 INFO:tasks.rados.rados.0.vpm137.stdout:3425: done (3 left) 2014-08-08T10:49:19.658 INFO:tasks.rados.rados.0.vpm137.stdout:3429: done (2 left) 2014-08-08T10:49:19.658 INFO:tasks.rados.rados.0.vpm137.stdout:3431: write oid 358 current snap is 0 2014-08-08T10:49:19.658 INFO:tasks.rados.rados.0.vpm137.stdout:3431: seq_num 2114 ranges {672212=605229,1711819=160393} 2014-08-08T10:49:19.659 INFO:tasks.rados.rados.0.vpm137.stdout:3431: writing vpm1375878-358 from 672212 to 1277441 tid 1 2014-08-08T10:49:19.659 INFO:tasks.rados.rados.0.vpm137.stdout:3431: writing vpm1375878-358 from 1711819 to 1872212 tid 2 2014-08-08T10:49:19.659 INFO:tasks.rados.rados.0.vpm137.stdout:3432: delete oid 369 current snap is 0 2014-08-08T10:49:19.659 INFO:tasks.rados.rados.0.vpm137.stdout:3428: finishing write tid 1 to vpm1375878-258 2014-08-08T10:49:19.800 INFO:tasks.rados.rados.0.vpm137.stdout:3428: finishing write tid 2 to vpm1375878-258 2014-08-08T10:49:19.811 INFO:tasks.rados.rados.0.vpm137.stdout:3428: finishing write tid 4 to vpm1375878-258 2014-08-08T10:49:19.819 INFO:tasks.rados.rados.0.vpm137.stdout:3428: finishing write tid 5 to vpm1375878-258 2014-08-08T10:49:19.820 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 258 v 368 (ObjNum 2112 snap 0 seq_num 2112) dirty exists 2014-08-08T10:49:19.830 INFO:tasks.rados.rados.0.vpm137.stdout:3430: finishing write tid 1 to vpm1375878-311 2014-08-08T10:49:19.831 INFO:tasks.rados.rados.0.vpm137.stdout:3430: finishing write tid 2 to vpm1375878-311 2014-08-08T10:49:19.832 INFO:tasks.rados.rados.0.vpm137.stdout:3430: finishing write tid 4 to vpm1375878-311 2014-08-08T10:49:19.833 INFO:tasks.rados.rados.0.vpm137.stdout:3431: finishing write tid 1 to vpm1375878-358 2014-08-08T10:49:19.833 INFO:tasks.rados.rados.0.vpm137.stdout:3431: finishing write tid 2 to vpm1375878-358 2014-08-08T10:49:19.849 INFO:tasks.rados.rados.0.vpm137.stdout:3431: finishing write tid 4 to vpm1375878-358 2014-08-08T10:49:19.849 INFO:tasks.rados.rados.0.vpm137.stdout:3428: done (3 left) 2014-08-08T10:49:19.849 INFO:tasks.rados.rados.0.vpm137.stdout:3432: done (2 left) 2014-08-08T10:49:19.849 INFO:tasks.rados.rados.0.vpm137.stdout:3433: read oid 265 snap -1 2014-08-08T10:49:19.849 INFO:tasks.rados.rados.0.vpm137.stdout:3434: write oid 161 current snap is 0 2014-08-08T10:49:19.850 INFO:tasks.rados.rados.0.vpm137.stdout:3434: seq_num 2116 ranges {677199=583910,1752564=592967,2869787=530282,3877198=1} 2014-08-08T10:49:19.882 INFO:tasks.rados.rados.0.vpm137.stdout:3434: writing vpm1375878-161 from 677199 to 1261109 tid 1 2014-08-08T10:49:19.927 INFO:tasks.rados.rados.0.vpm137.stdout:3434: writing vpm1375878-161 from 1752564 to 2345531 tid 2 2014-08-08T10:49:19.965 INFO:tasks.rados.rados.0.vpm137.stdout:3434: writing vpm1375878-161 from 2869787 to 3400069 tid 3 2014-08-08T10:49:19.974 INFO:tasks.rados.rados.0.vpm137.stdout:3434: writing vpm1375878-161 from 3877198 to 3877199 tid 4 2014-08-08T10:49:19.974 INFO:tasks.rados.rados.0.vpm137.stdout:3435: write oid 225 current snap is 0 2014-08-08T10:49:19.975 INFO:tasks.rados.rados.0.vpm137.stdout:3435: seq_num 2117 ranges {772429=707138,2016072=663162,3372836=199593} 2014-08-08T10:49:20.016 INFO:tasks.rados.rados.0.vpm137.stdout:3435: writing vpm1375878-225 from 772429 to 1479567 tid 1 2014-08-08T10:49:20.045 INFO:tasks.rados.rados.0.vpm137.stdout:3435: writing vpm1375878-225 from 2016072 to 2679234 tid 2 2014-08-08T10:49:20.058 INFO:tasks.rados.rados.0.vpm137.stdout:3435: writing vpm1375878-225 from 3372836 to 3572429 tid 3 2014-08-08T10:49:20.060 INFO:tasks.rados.rados.0.vpm137.stdout:3436: read oid 169 snap -1 2014-08-08T10:49:20.060 INFO:tasks.rados.rados.0.vpm137.stdout:3437: delete oid 482 current snap is 0 2014-08-08T10:49:20.105 INFO:tasks.rados.rados.0.vpm137.stdout:3436: expect (ObjNum 1982 snap 0 seq_num 1982) 2014-08-08T10:49:21.069 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:21.292 INFO:tasks.rados.rados.0.vpm137.stdout:3436: done (6 left) 2014-08-08T10:49:21.292 INFO:tasks.rados.rados.0.vpm137.stdout:3437: done (5 left) 2014-08-08T10:49:21.293 INFO:tasks.rados.rados.0.vpm137.stdout:3438: read oid 61 snap -1 2014-08-08T10:49:21.293 INFO:tasks.rados.rados.0.vpm137.stdout:3439: delete oid 117 current snap is 0 2014-08-08T10:49:21.294 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 1 to vpm1375878-161 2014-08-08T10:49:21.294 INFO:tasks.rados.rados.0.vpm137.stdout:3433: expect (ObjNum 1990 snap 0 seq_num 1990) 2014-08-08T10:49:21.899 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 2 to vpm1375878-161 2014-08-08T10:49:21.900 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 3 to vpm1375878-161 2014-08-08T10:49:21.900 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 4 to vpm1375878-161 2014-08-08T10:49:21.901 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 6 to vpm1375878-161 2014-08-08T10:49:21.901 INFO:tasks.rados.rados.0.vpm137.stdout:3435: finishing write tid 1 to vpm1375878-225 2014-08-08T10:49:21.901 INFO:tasks.rados.rados.0.vpm137.stdout:3435: finishing write tid 2 to vpm1375878-225 2014-08-08T10:49:21.902 INFO:tasks.rados.rados.0.vpm137.stdout:3435: finishing write tid 3 to vpm1375878-225 2014-08-08T10:49:21.902 INFO:tasks.rados.rados.0.vpm137.stdout:3435: finishing write tid 5 to vpm1375878-225 2014-08-08T10:49:21.903 INFO:tasks.rados.rados.0.vpm137.stdout:3430: finishing write tid 5 to vpm1375878-311 2014-08-08T10:49:21.903 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 311 v 482 (ObjNum 2113 snap 0 seq_num 2113) dirty exists 2014-08-08T10:49:21.904 INFO:tasks.rados.rados.0.vpm137.stdout:3431: finishing write tid 5 to vpm1375878-358 2014-08-08T10:49:21.904 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 358 v 404 (ObjNum 2114 snap 0 seq_num 2114) dirty exists 2014-08-08T10:49:21.904 INFO:tasks.rados.rados.0.vpm137.stdout:3434: finishing write tid 7 to vpm1375878-161 2014-08-08T10:49:21.905 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 161 v 409 (ObjNum 2116 snap 0 seq_num 2116) dirty exists 2014-08-08T10:49:21.905 INFO:tasks.rados.rados.0.vpm137.stdout:3435: finishing write tid 6 to vpm1375878-225 2014-08-08T10:49:21.906 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 225 v 312 (ObjNum 2117 snap 0 seq_num 2117) dirty exists 2014-08-08T10:49:21.942 INFO:tasks.rados.rados.0.vpm137.stdout:3430: done (6 left) 2014-08-08T10:49:21.943 INFO:tasks.rados.rados.0.vpm137.stdout:3431: done (5 left) 2014-08-08T10:49:21.943 INFO:tasks.rados.rados.0.vpm137.stdout:3433: done (4 left) 2014-08-08T10:49:21.943 INFO:tasks.rados.rados.0.vpm137.stdout:3434: done (3 left) 2014-08-08T10:49:21.943 INFO:tasks.rados.rados.0.vpm137.stdout:3435: done (2 left) 2014-08-08T10:49:21.944 INFO:tasks.rados.rados.0.vpm137.stdout:3439: done (1 left) 2014-08-08T10:49:21.944 INFO:tasks.rados.rados.0.vpm137.stdout:3440: write oid 486 current snap is 0 2014-08-08T10:49:21.944 INFO:tasks.rados.rados.0.vpm137.stdout:3440: seq_num 2120 ranges {472646=658647,1272645=1} 2014-08-08T10:49:21.969 INFO:tasks.rados.rados.0.vpm137.stdout:3440: writing vpm1375878-486 from 472646 to 1131293 tid 1 2014-08-08T10:49:21.972 INFO:tasks.rados.rados.0.vpm137.stdout:3440: writing vpm1375878-486 from 1272645 to 1272646 tid 2 2014-08-08T10:49:21.974 INFO:tasks.rados.rados.0.vpm137.stdout:3441: read oid 396 snap -1 2014-08-08T10:49:21.974 INFO:tasks.rados.rados.0.vpm137.stdout:3442: read oid 434 snap -1 2014-08-08T10:49:21.975 INFO:tasks.rados.rados.0.vpm137.stdout:3443: write oid 114 current snap is 0 2014-08-08T10:49:21.975 INFO:tasks.rados.rados.0.vpm137.stdout:3443: seq_num 2121 ranges {568900=400000} 2014-08-08T10:49:21.998 INFO:tasks.rados.rados.0.vpm137.stdout:3443: writing vpm1375878-114 from 568900 to 968900 tid 1 2014-08-08T10:49:22.000 INFO:tasks.rados.rados.0.vpm137.stdout:3438: expect (ObjNum 1817 snap 0 seq_num 1817) 2014-08-08T10:49:22.594 INFO:tasks.rados.rados.0.vpm137.stdout:3438: done (4 left) 2014-08-08T10:49:22.595 INFO:tasks.rados.rados.0.vpm137.stdout:3444: write oid 62 current snap is 0 2014-08-08T10:49:22.595 INFO:tasks.rados.rados.0.vpm137.stdout:3444: seq_num 2122 ranges {762306=754704,2213741=548565} 2014-08-08T10:49:22.635 INFO:tasks.rados.rados.0.vpm137.stdout:3444: writing vpm1375878-62 from 762306 to 1517010 tid 1 2014-08-08T10:49:22.667 INFO:tasks.rados.rados.0.vpm137.stdout:3444: writing vpm1375878-62 from 2213741 to 2762306 tid 2 2014-08-08T10:49:22.671 INFO:tasks.rados.rados.0.vpm137.stdout:3440: finishing write tid 1 to vpm1375878-486 2014-08-08T10:49:22.671 INFO:tasks.rados.rados.0.vpm137.stdout:3440: finishing write tid 2 to vpm1375878-486 2014-08-08T10:49:22.671 INFO:tasks.rados.rados.0.vpm137.stdout:3440: finishing write tid 4 to vpm1375878-486 2014-08-08T10:49:22.672 INFO:tasks.rados.rados.0.vpm137.stdout:3443: finishing write tid 1 to vpm1375878-114 2014-08-08T10:49:22.672 INFO:tasks.rados.rados.0.vpm137.stdout:3440: finishing write tid 5 to vpm1375878-486 2014-08-08T10:49:22.672 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 486 v 343 (ObjNum 2120 snap 0 seq_num 2120) dirty exists 2014-08-08T10:49:22.672 INFO:tasks.rados.rados.0.vpm137.stdout:3443: finishing write tid 3 to vpm1375878-114 2014-08-08T10:49:22.673 INFO:tasks.rados.rados.0.vpm137.stdout:3443: finishing write tid 4 to vpm1375878-114 2014-08-08T10:49:22.673 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 114 v 333 (ObjNum 2121 snap 0 seq_num 2121) dirty exists 2014-08-08T10:49:22.673 INFO:tasks.rados.rados.0.vpm137.stdout:3444: finishing write tid 1 to vpm1375878-62 2014-08-08T10:49:22.673 INFO:tasks.rados.rados.0.vpm137.stdout:3440: done (4 left) 2014-08-08T10:49:22.674 INFO:tasks.rados.rados.0.vpm137.stdout:3441: done (3 left) 2014-08-08T10:49:22.674 INFO:tasks.rados.rados.0.vpm137.stdout:3442: done (2 left) 2014-08-08T10:49:22.675 INFO:tasks.rados.rados.0.vpm137.stdout:3443: done (1 left) 2014-08-08T10:49:22.675 INFO:tasks.rados.rados.0.vpm137.stdout:3445: write oid 318 current snap is 0 2014-08-08T10:49:22.675 INFO:tasks.rados.rados.0.vpm137.stdout:3445: seq_num 2123 ranges {457536=761580,1678273=587181,2457535=1} 2014-08-08T10:49:22.702 INFO:tasks.rados.rados.0.vpm137.stdout:3445: writing vpm1375878-318 from 457536 to 1219116 tid 1 2014-08-08T10:49:22.728 INFO:tasks.rados.rados.0.vpm137.stdout:3445: writing vpm1375878-318 from 1678273 to 2265454 tid 2 2014-08-08T10:49:22.731 INFO:tasks.rados.rados.0.vpm137.stdout:3445: writing vpm1375878-318 from 2457535 to 2457536 tid 3 2014-08-08T10:49:22.733 INFO:tasks.rados.rados.0.vpm137.stdout:3444: finishing write tid 2 to vpm1375878-62 2014-08-08T10:49:22.733 INFO:tasks.rados.rados.0.vpm137.stdout:3444: finishing write tid 4 to vpm1375878-62 2014-08-08T10:49:22.733 INFO:tasks.rados.rados.0.vpm137.stdout:3444: finishing write tid 5 to vpm1375878-62 2014-08-08T10:49:22.733 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 62 v 313 (ObjNum 2122 snap 0 seq_num 2122) dirty exists 2014-08-08T10:49:22.733 INFO:tasks.rados.rados.0.vpm137.stdout:3444: done (1 left) 2014-08-08T10:49:22.734 INFO:tasks.rados.rados.0.vpm137.stdout:3446: write oid 291 current snap is 0 2014-08-08T10:49:22.734 INFO:tasks.rados.rados.0.vpm137.stdout:3446: seq_num 2124 ranges {670141=581961,2005909=657376,2670140=1} 2014-08-08T10:49:22.760 INFO:tasks.rados.rados.0.vpm137.stdout:3446: writing vpm1375878-291 from 670141 to 1252102 tid 1 2014-08-08T10:49:22.796 INFO:tasks.rados.rados.0.vpm137.stdout:3446: writing vpm1375878-291 from 2005909 to 2663285 tid 2 2014-08-08T10:49:22.798 INFO:tasks.rados.rados.0.vpm137.stdout:3446: writing vpm1375878-291 from 2670140 to 2670141 tid 3 2014-08-08T10:49:22.799 INFO:tasks.rados.rados.0.vpm137.stdout:3445: finishing write tid 1 to vpm1375878-318 2014-08-08T10:49:22.800 INFO:tasks.rados.rados.0.vpm137.stdout:3445: finishing write tid 2 to vpm1375878-318 2014-08-08T10:49:22.800 INFO:tasks.rados.rados.0.vpm137.stdout:3445: finishing write tid 3 to vpm1375878-318 2014-08-08T10:49:22.800 INFO:tasks.rados.rados.0.vpm137.stdout:3445: finishing write tid 5 to vpm1375878-318 2014-08-08T10:49:22.800 INFO:tasks.rados.rados.0.vpm137.stdout:3445: finishing write tid 6 to vpm1375878-318 2014-08-08T10:49:22.801 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 318 v 316 (ObjNum 2123 snap 0 seq_num 2123) dirty exists 2014-08-08T10:49:22.801 INFO:tasks.rados.rados.0.vpm137.stdout:3446: finishing write tid 1 to vpm1375878-291 2014-08-08T10:49:22.801 INFO:tasks.rados.rados.0.vpm137.stdout:3445: done (1 left) 2014-08-08T10:49:22.801 INFO:tasks.rados.rados.0.vpm137.stdout:3447: write oid 463 current snap is 0 2014-08-08T10:49:22.801 INFO:tasks.rados.rados.0.vpm137.stdout:3447: seq_num 2125 ranges {462522=1} 2014-08-08T10:49:22.804 INFO:tasks.rados.rados.0.vpm137.stdout:3447: writing vpm1375878-463 from 462522 to 462523 tid 1 2014-08-08T10:49:22.805 INFO:tasks.rados.rados.0.vpm137.stdout:3448: read oid 332 snap -1 2014-08-08T10:49:22.805 INFO:tasks.rados.rados.0.vpm137.stdout:3449: read oid 12 snap -1 2014-08-08T10:49:22.806 INFO:tasks.rados.rados.0.vpm137.stdout:3450: read oid 108 snap -1 2014-08-08T10:49:22.807 INFO:tasks.rados.rados.0.vpm137.stdout:3451: write oid 481 current snap is 0 2014-08-08T10:49:22.807 INFO:tasks.rados.rados.0.vpm137.stdout:3451: seq_num 2126 ranges {158776=1} 2014-08-08T10:49:22.808 INFO:tasks.rados.rados.0.vpm137.stdout:3451: writing vpm1375878-481 from 158776 to 158777 tid 1 2014-08-08T10:49:22.808 INFO:tasks.rados.rados.0.vpm137.stdout:3452: write oid 403 current snap is 0 2014-08-08T10:49:22.809 INFO:tasks.rados.rados.0.vpm137.stdout:3452: seq_num 2127 ranges {655031=684894,1795858=654458,2913666=650865,3855030=1} 2014-08-08T10:49:22.835 INFO:tasks.rados.rados.0.vpm137.stdout:3452: writing vpm1375878-403 from 655031 to 1339925 tid 1 2014-08-08T10:49:22.866 INFO:tasks.rados.rados.0.vpm137.stdout:3452: writing vpm1375878-403 from 1795858 to 2450316 tid 2 2014-08-08T10:49:22.899 INFO:tasks.rados.rados.0.vpm137.stdout:3452: writing vpm1375878-403 from 2913666 to 3564531 tid 3 2014-08-08T10:49:22.902 INFO:tasks.rados.rados.0.vpm137.stdout:3452: writing vpm1375878-403 from 3855030 to 3855031 tid 4 2014-08-08T10:49:22.903 INFO:tasks.rados.rados.0.vpm137.stdout:3446: finishing write tid 2 to vpm1375878-291 2014-08-08T10:49:22.903 INFO:tasks.rados.rados.0.vpm137.stdout:3448: expect (ObjNum 1438 snap 0 seq_num 1438) 2014-08-08T10:49:24.096 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:24.304 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:24.319 INFO:tasks.rados.rados.0.vpm137.stdout:3448: done (6 left) 2014-08-08T10:49:24.320 INFO:tasks.rados.rados.0.vpm137.stdout:3453: read oid 420 snap -1 2014-08-08T10:49:24.321 INFO:tasks.rados.rados.0.vpm137.stdout:3454: write oid 326 current snap is 0 2014-08-08T10:49:24.321 INFO:tasks.rados.rados.0.vpm137.stdout:3454: seq_num 2128 ranges {562740=657722,1838821=323919} 2014-08-08T10:49:24.362 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:24.363 INFO:tasks.rados.rados.0.vpm137.stdout:3454: writing vpm1375878-326 from 562740 to 1220462 tid 1 2014-08-08T10:49:24.373 INFO:tasks.rados.rados.0.vpm137.stdout:3454: writing vpm1375878-326 from 1838821 to 2162740 tid 2 2014-08-08T10:49:24.374 INFO:tasks.rados.rados.0.vpm137.stdout:3449: expect (ObjNum 1069 snap 0 seq_num 1069) 2014-08-08T10:49:26.007 INFO:tasks.rados.rados.0.vpm137.stdout:3449: done (7 left) 2014-08-08T10:49:26.007 INFO:tasks.rados.rados.0.vpm137.stdout:3450: done (6 left) 2014-08-08T10:49:26.008 INFO:tasks.rados.rados.0.vpm137.stdout:3455: write oid 443 current snap is 0 2014-08-08T10:49:26.008 INFO:tasks.rados.rados.0.vpm137.stdout:3455: seq_num 2129 ranges {658994=780950,1858993=1} 2014-08-08T10:49:26.086 INFO:tasks.rados.rados.0.vpm137.stdout:3455: writing vpm1375878-443 from 658994 to 1439944 tid 1 2014-08-08T10:49:26.100 INFO:tasks.rados.rados.0.vpm137.stdout:3455: writing vpm1375878-443 from 1858993 to 1858994 tid 2 2014-08-08T10:49:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:3446: finishing write tid 3 to vpm1375878-291 2014-08-08T10:49:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:3446: finishing write tid 5 to vpm1375878-291 2014-08-08T10:49:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3447: finishing write tid 1 to vpm1375878-463 2014-08-08T10:49:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3447: finishing write tid 3 to vpm1375878-463 2014-08-08T10:49:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3451: finishing write tid 1 to vpm1375878-481 2014-08-08T10:49:26.104 INFO:tasks.rados.rados.0.vpm137.stdout:3451: finishing write tid 3 to vpm1375878-481 2014-08-08T10:49:26.104 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 1 to vpm1375878-403 2014-08-08T10:49:26.104 INFO:tasks.rados.rados.0.vpm137.stdout:3447: finishing write tid 4 to vpm1375878-463 2014-08-08T10:49:26.104 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 463 v 318 (ObjNum 2125 snap 0 seq_num 2125) dirty exists 2014-08-08T10:49:26.104 INFO:tasks.rados.rados.0.vpm137.stdout:3446: finishing write tid 6 to vpm1375878-291 2014-08-08T10:49:26.105 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 291 v 317 (ObjNum 2124 snap 0 seq_num 2124) dirty exists 2014-08-08T10:49:26.105 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 2 to vpm1375878-403 2014-08-08T10:49:26.105 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 3 to vpm1375878-403 2014-08-08T10:49:26.105 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 4 to vpm1375878-403 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:3451: finishing write tid 4 to vpm1375878-481 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 481 v 484 (ObjNum 2126 snap 0 seq_num 2126) dirty exists 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 6 to vpm1375878-403 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:3452: finishing write tid 7 to vpm1375878-403 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 403 v 323 (ObjNum 2127 snap 0 seq_num 2127) dirty exists 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:3454: finishing write tid 1 to vpm1375878-326 2014-08-08T10:49:26.106 INFO:tasks.rados.rados.0.vpm137.stdout:3454: finishing write tid 2 to vpm1375878-326 2014-08-08T10:49:26.107 INFO:tasks.rados.rados.0.vpm137.stdout:3454: finishing write tid 4 to vpm1375878-326 2014-08-08T10:49:26.107 INFO:tasks.rados.rados.0.vpm137.stdout:3454: finishing write tid 5 to vpm1375878-326 2014-08-08T10:49:26.107 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 326 v 383 (ObjNum 2128 snap 0 seq_num 2128) dirty exists 2014-08-08T10:49:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3446: done (6 left) 2014-08-08T10:49:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3447: done (5 left) 2014-08-08T10:49:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3451: done (4 left) 2014-08-08T10:49:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3452: done (3 left) 2014-08-08T10:49:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3453: done (2 left) 2014-08-08T10:49:26.109 INFO:tasks.rados.rados.0.vpm137.stdout:3454: done (1 left) 2014-08-08T10:49:26.109 INFO:tasks.rados.rados.0.vpm137.stdout:3456: write oid 120 current snap is 0 2014-08-08T10:49:26.109 INFO:tasks.rados.rados.0.vpm137.stdout:3456: seq_num 2130 ranges {754224=752755,1554223=1} 2014-08-08T10:49:26.133 INFO:tasks.rados.rados.0.vpm137.stdout:3456: writing vpm1375878-120 from 754224 to 1506979 tid 1 2014-08-08T10:49:26.136 INFO:tasks.rados.rados.0.vpm137.stdout:3456: writing vpm1375878-120 from 1554223 to 1554224 tid 2 2014-08-08T10:49:26.136 INFO:tasks.rados.rados.0.vpm137.stdout:3457: read oid 494 snap -1 2014-08-08T10:49:26.137 INFO:tasks.rados.rados.0.vpm137.stdout:3458: read oid 356 snap -1 2014-08-08T10:49:26.137 INFO:tasks.rados.rados.0.vpm137.stdout:3459: write oid 254 current snap is 0 2014-08-08T10:49:26.138 INFO:tasks.rados.rados.0.vpm137.stdout:3459: seq_num 2131 ranges {547629=759631,2028769=535238,3263270=84359} 2014-08-08T10:49:26.169 INFO:tasks.rados.rados.0.vpm137.stdout:3459: writing vpm1375878-254 from 547629 to 1307260 tid 1 2014-08-08T10:49:26.194 INFO:tasks.rados.rados.0.vpm137.stdout:3459: writing vpm1375878-254 from 2028769 to 2564007 tid 2 2014-08-08T10:49:26.203 INFO:tasks.rados.rados.0.vpm137.stdout:3459: writing vpm1375878-254 from 3263270 to 3347629 tid 3 2014-08-08T10:49:26.203 INFO:tasks.rados.rados.0.vpm137.stdout:3455: finishing write tid 1 to vpm1375878-443 2014-08-08T10:49:26.204 INFO:tasks.rados.rados.0.vpm137.stdout:3455: finishing write tid 2 to vpm1375878-443 2014-08-08T10:49:26.204 INFO:tasks.rados.rados.0.vpm137.stdout:3460: write oid 55 current snap is 0 2014-08-08T10:49:26.205 INFO:tasks.rados.rados.0.vpm137.stdout:3460: seq_num 2132 ranges {643883=732459,2143926=756858,3043882=1} 2014-08-08T10:49:26.233 INFO:tasks.rados.rados.0.vpm137.stdout:3460: writing vpm1375878-55 from 643883 to 1376342 tid 1 2014-08-08T10:49:26.266 INFO:tasks.rados.rados.0.vpm137.stdout:3460: writing vpm1375878-55 from 2143926 to 2900784 tid 2 2014-08-08T10:49:26.268 INFO:tasks.rados.rados.0.vpm137.stdout:3460: writing vpm1375878-55 from 3043882 to 3043883 tid 3 2014-08-08T10:49:26.268 INFO:tasks.rados.rados.0.vpm137.stdout:3457: done (5 left) 2014-08-08T10:49:26.269 INFO:tasks.rados.rados.0.vpm137.stdout:3461: read oid 224 snap -1 2014-08-08T10:49:26.269 INFO:tasks.rados.rados.0.vpm137.stdout:3462: write oid 243 current snap is 0 2014-08-08T10:49:26.269 INFO:tasks.rados.rados.0.vpm137.stdout:3462: seq_num 2133 ranges {455465=455688,1688715=426029,2848827=406638} 2014-08-08T10:49:26.289 INFO:tasks.rados.rados.0.vpm137.stdout:3462: writing vpm1375878-243 from 455465 to 911153 tid 1 2014-08-08T10:49:26.309 INFO:tasks.rados.rados.0.vpm137.stdout:3462: writing vpm1375878-243 from 1688715 to 2114744 tid 2 2014-08-08T10:49:26.338 INFO:tasks.rados.rados.0.vpm137.stdout:3462: writing vpm1375878-243 from 2848827 to 3255465 tid 3 2014-08-08T10:49:26.339 INFO:tasks.rados.rados.0.vpm137.stdout:3461: expect (ObjNum 2096 snap 0 seq_num 2096) 2014-08-08T10:49:26.979 INFO:tasks.rados.rados.0.vpm137.stdout:3461: done (6 left) 2014-08-08T10:49:26.979 INFO:tasks.rados.rados.0.vpm137.stdout:3463: write oid 382 current snap is 0 2014-08-08T10:49:26.979 INFO:tasks.rados.rados.0.vpm137.stdout:3463: seq_num 2134 ranges {294=1} 2014-08-08T10:49:26.990 INFO:tasks.rados.rados.0.vpm137.stdout:3463: writing vpm1375878-382 from 294 to 295 tid 1 2014-08-08T10:49:26.991 INFO:tasks.rados.rados.0.vpm137.stdout:3455: finishing write tid 4 to vpm1375878-443 2014-08-08T10:49:26.991 INFO:tasks.rados.rados.0.vpm137.stdout:3458: expect (ObjNum 1554 snap 0 seq_num 1554) 2014-08-08T10:49:27.129 INFO:tasks.rados.rados.0.vpm137.stdout:3458: done (6 left) 2014-08-08T10:49:27.130 INFO:tasks.rados.rados.0.vpm137.stdout:3464: write oid 279 current snap is 0 2014-08-08T10:49:27.130 INFO:tasks.rados.rados.0.vpm137.stdout:3464: seq_num 2135 ranges {744099=1} 2014-08-08T10:49:27.135 INFO:tasks.rados.rados.0.vpm137.stdout:3464: writing vpm1375878-279 from 744099 to 744100 tid 1 2014-08-08T10:49:27.136 INFO:tasks.rados.rados.0.vpm137.stdout:3455: finishing write tid 5 to vpm1375878-443 2014-08-08T10:49:27.136 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 443 v 376 (ObjNum 2129 snap 0 seq_num 2129) dirty exists 2014-08-08T10:49:27.137 INFO:tasks.rados.rados.0.vpm137.stdout:3456: finishing write tid 1 to vpm1375878-120 2014-08-08T10:49:27.137 INFO:tasks.rados.rados.0.vpm137.stdout:3456: finishing write tid 2 to vpm1375878-120 2014-08-08T10:49:27.137 INFO:tasks.rados.rados.0.vpm137.stdout:3456: finishing write tid 4 to vpm1375878-120 2014-08-08T10:49:27.137 INFO:tasks.rados.rados.0.vpm137.stdout:3459: finishing write tid 1 to vpm1375878-254 2014-08-08T10:49:27.137 INFO:tasks.rados.rados.0.vpm137.stdout:3459: finishing write tid 2 to vpm1375878-254 2014-08-08T10:49:27.138 INFO:tasks.rados.rados.0.vpm137.stdout:3459: finishing write tid 3 to vpm1375878-254 2014-08-08T10:49:27.138 INFO:tasks.rados.rados.0.vpm137.stdout:3459: finishing write tid 5 to vpm1375878-254 2014-08-08T10:49:27.138 INFO:tasks.rados.rados.0.vpm137.stdout:3460: finishing write tid 1 to vpm1375878-55 2014-08-08T10:49:27.138 INFO:tasks.rados.rados.0.vpm137.stdout:3460: finishing write tid 2 to vpm1375878-55 2014-08-08T10:49:27.139 INFO:tasks.rados.rados.0.vpm137.stdout:3460: finishing write tid 3 to vpm1375878-55 2014-08-08T10:49:27.139 INFO:tasks.rados.rados.0.vpm137.stdout:3460: finishing write tid 5 to vpm1375878-55 2014-08-08T10:49:27.139 INFO:tasks.rados.rados.0.vpm137.stdout:3462: finishing write tid 1 to vpm1375878-243 2014-08-08T10:49:27.139 INFO:tasks.rados.rados.0.vpm137.stdout:3462: finishing write tid 2 to vpm1375878-243 2014-08-08T10:49:27.139 INFO:tasks.rados.rados.0.vpm137.stdout:3462: finishing write tid 3 to vpm1375878-243 2014-08-08T10:49:27.140 INFO:tasks.rados.rados.0.vpm137.stdout:3462: finishing write tid 5 to vpm1375878-243 2014-08-08T10:49:27.140 INFO:tasks.rados.rados.0.vpm137.stdout:3456: finishing write tid 5 to vpm1375878-120 2014-08-08T10:49:27.140 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 120 v 379 (ObjNum 2130 snap 0 seq_num 2130) dirty exists 2014-08-08T10:49:27.140 INFO:tasks.rados.rados.0.vpm137.stdout:3459: finishing write tid 6 to vpm1375878-254 2014-08-08T10:49:27.140 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 254 v 421 (ObjNum 2131 snap 0 seq_num 2131) dirty exists 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:3460: finishing write tid 6 to vpm1375878-55 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 55 v 383 (ObjNum 2132 snap 0 seq_num 2132) dirty exists 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:3462: finishing write tid 6 to vpm1375878-243 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 243 v 392 (ObjNum 2133 snap 0 seq_num 2133) dirty exists 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:3463: finishing write tid 1 to vpm1375878-382 2014-08-08T10:49:27.141 INFO:tasks.rados.rados.0.vpm137.stdout:3463: finishing write tid 3 to vpm1375878-382 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3463: finishing write tid 4 to vpm1375878-382 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 382 v 411 (ObjNum 2134 snap 0 seq_num 2134) dirty exists 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3455: done (6 left) 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3456: done (5 left) 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3459: done (4 left) 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3460: done (3 left) 2014-08-08T10:49:27.142 INFO:tasks.rados.rados.0.vpm137.stdout:3462: done (2 left) 2014-08-08T10:49:27.143 INFO:tasks.rados.rados.0.vpm137.stdout:3463: done (1 left) 2014-08-08T10:49:27.143 INFO:tasks.rados.rados.0.vpm137.stdout:3465: read oid 186 snap -1 2014-08-08T10:49:27.143 INFO:tasks.rados.rados.0.vpm137.stdout:3466: write oid 236 current snap is 0 2014-08-08T10:49:27.143 INFO:tasks.rados.rados.0.vpm137.stdout:3466: seq_num 2136 ranges {439329=1} 2014-08-08T10:49:27.143 INFO:tasks.rados.rados.0.vpm137.stdout:3466: writing vpm1375878-236 from 439329 to 439330 tid 1 2014-08-08T10:49:27.144 INFO:tasks.rados.rados.0.vpm137.stdout:3467: write oid 397 current snap is 0 2014-08-08T10:49:27.144 INFO:tasks.rados.rados.0.vpm137.stdout:3467: seq_num 2137 ranges {500512=779001,1700511=1} 2014-08-08T10:49:27.186 INFO:tasks.rados.rados.0.vpm137.stdout:3467: writing vpm1375878-397 from 500512 to 1279513 tid 1 2014-08-08T10:49:27.194 INFO:tasks.rados.rados.0.vpm137.stdout:3467: writing vpm1375878-397 from 1700511 to 1700512 tid 2 2014-08-08T10:49:27.195 INFO:tasks.rados.rados.0.vpm137.stdout:3468: write oid 161 current snap is 0 2014-08-08T10:49:27.195 INFO:tasks.rados.rados.0.vpm137.stdout:3468: seq_num 2138 ranges {444318=750805,1884335=559983} 2014-08-08T10:49:27.238 INFO:tasks.rados.rados.0.vpm137.stdout:3468: writing vpm1375878-161 from 444318 to 1195123 tid 1 2014-08-08T10:49:27.265 INFO:tasks.rados.rados.0.vpm137.stdout:3468: writing vpm1375878-161 from 1884335 to 2444318 tid 2 2014-08-08T10:49:27.267 INFO:tasks.rados.rados.0.vpm137.stdout:3465: expect (ObjNum 1504 snap 0 seq_num 1504) 2014-08-08T10:49:27.578 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 825876813, u'bytes_used': 1778593792, u'op_per_sec': 12, u'version': 1663, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1319269, u'write_bytes_sec': 1456016, u'bytes_avail': 427508420608}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:27.578 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:27.776 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:27.777 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:27.972 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:29.389 INFO:tasks.rados.rados.0.vpm137.stdout:3465: done (4 left) 2014-08-08T10:49:29.389 INFO:tasks.rados.rados.0.vpm137.stdout:3469: read oid 472 snap -1 2014-08-08T10:49:29.390 INFO:tasks.rados.rados.0.vpm137.stdout:3470: read oid 382 snap -1 2014-08-08T10:49:29.390 INFO:tasks.rados.rados.0.vpm137.stdout:3471: write oid 420 current snap is 0 2014-08-08T10:49:29.391 INFO:tasks.rados.rados.0.vpm137.stdout:3471: seq_num 2139 ranges {540571=758706,1731715=408856} 2014-08-08T10:49:29.448 INFO:tasks.rados.rados.0.vpm137.stdout:3471: writing vpm1375878-420 from 540571 to 1299277 tid 1 2014-08-08T10:49:29.485 INFO:tasks.rados.rados.0.vpm137.stdout:3471: writing vpm1375878-420 from 1731715 to 2140571 tid 2 2014-08-08T10:49:29.487 INFO:tasks.rados.rados.0.vpm137.stdout:3464: finishing write tid 1 to vpm1375878-279 2014-08-08T10:49:29.487 INFO:tasks.rados.rados.0.vpm137.stdout:3464: finishing write tid 3 to vpm1375878-279 2014-08-08T10:49:29.488 INFO:tasks.rados.rados.0.vpm137.stdout:3464: finishing write tid 4 to vpm1375878-279 2014-08-08T10:49:29.489 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 279 v 325 (ObjNum 2135 snap 0 seq_num 2135) dirty exists 2014-08-08T10:49:29.489 INFO:tasks.rados.rados.0.vpm137.stdout:3466: finishing write tid 1 to vpm1375878-236 2014-08-08T10:49:29.489 INFO:tasks.rados.rados.0.vpm137.stdout:3466: finishing write tid 3 to vpm1375878-236 2014-08-08T10:49:29.489 INFO:tasks.rados.rados.0.vpm137.stdout:3466: finishing write tid 4 to vpm1375878-236 2014-08-08T10:49:29.491 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 236 v 295 (ObjNum 2136 snap 0 seq_num 2136) dirty exists 2014-08-08T10:49:29.491 INFO:tasks.rados.rados.0.vpm137.stdout:3467: finishing write tid 1 to vpm1375878-397 2014-08-08T10:49:29.491 INFO:tasks.rados.rados.0.vpm137.stdout:3467: finishing write tid 2 to vpm1375878-397 2014-08-08T10:49:29.491 INFO:tasks.rados.rados.0.vpm137.stdout:3467: finishing write tid 4 to vpm1375878-397 2014-08-08T10:49:29.492 INFO:tasks.rados.rados.0.vpm137.stdout:3467: finishing write tid 5 to vpm1375878-397 2014-08-08T10:49:29.492 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 397 v 421 (ObjNum 2137 snap 0 seq_num 2137) dirty exists 2014-08-08T10:49:29.492 INFO:tasks.rados.rados.0.vpm137.stdout:3468: finishing write tid 1 to vpm1375878-161 2014-08-08T10:49:29.493 INFO:tasks.rados.rados.0.vpm137.stdout:3468: finishing write tid 2 to vpm1375878-161 2014-08-08T10:49:29.493 INFO:tasks.rados.rados.0.vpm137.stdout:3468: finishing write tid 4 to vpm1375878-161 2014-08-08T10:49:29.493 INFO:tasks.rados.rados.0.vpm137.stdout:3468: finishing write tid 5 to vpm1375878-161 2014-08-08T10:49:29.493 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 161 v 414 (ObjNum 2138 snap 0 seq_num 2138) dirty exists 2014-08-08T10:49:29.494 INFO:tasks.rados.rados.0.vpm137.stdout:3469: expect (ObjNum 1614 snap 0 seq_num 1614) 2014-08-08T10:49:30.439 INFO:tasks.rados.rados.0.vpm137.stdout:3464: done (6 left) 2014-08-08T10:49:30.439 INFO:tasks.rados.rados.0.vpm137.stdout:3466: done (5 left) 2014-08-08T10:49:30.439 INFO:tasks.rados.rados.0.vpm137.stdout:3467: done (4 left) 2014-08-08T10:49:30.439 INFO:tasks.rados.rados.0.vpm137.stdout:3468: done (3 left) 2014-08-08T10:49:30.440 INFO:tasks.rados.rados.0.vpm137.stdout:3469: done (2 left) 2014-08-08T10:49:30.440 INFO:tasks.rados.rados.0.vpm137.stdout:3472: read oid 10 snap -1 2014-08-08T10:49:30.440 INFO:tasks.rados.rados.0.vpm137.stdout:3473: read oid 422 snap -1 2014-08-08T10:49:30.441 INFO:tasks.rados.rados.0.vpm137.stdout:3474: write oid 386 current snap is 0 2014-08-08T10:49:30.441 INFO:tasks.rados.rados.0.vpm137.stdout:3474: seq_num 2140 ranges {484377=730510,1809751=455315,2884376=1} 2014-08-08T10:49:30.487 INFO:tasks.rados.rados.0.vpm137.stdout:3474: writing vpm1375878-386 from 484377 to 1214887 tid 1 2014-08-08T10:49:30.528 INFO:tasks.rados.rados.0.vpm137.stdout:3474: writing vpm1375878-386 from 1809751 to 2265066 tid 2 2014-08-08T10:49:30.539 INFO:tasks.rados.rados.0.vpm137.stdout:3474: writing vpm1375878-386 from 2884376 to 2884377 tid 3 2014-08-08T10:49:30.540 INFO:tasks.rados.rados.0.vpm137.stdout:3470: expect (ObjNum 2134 snap 0 seq_num 2134) 2014-08-08T10:49:30.540 INFO:tasks.rados.rados.0.vpm137.stdout:3471: finishing write tid 1 to vpm1375878-420 2014-08-08T10:49:30.540 INFO:tasks.rados.rados.0.vpm137.stdout:3471: finishing write tid 2 to vpm1375878-420 2014-08-08T10:49:30.541 INFO:tasks.rados.rados.0.vpm137.stdout:3471: finishing write tid 4 to vpm1375878-420 2014-08-08T10:49:30.541 INFO:tasks.rados.rados.0.vpm137.stdout:3471: finishing write tid 5 to vpm1375878-420 2014-08-08T10:49:30.541 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 420 v 328 (ObjNum 2139 snap 0 seq_num 2139) dirty exists 2014-08-08T10:49:30.541 INFO:tasks.rados.rados.0.vpm137.stdout:3470: done (4 left) 2014-08-08T10:49:30.541 INFO:tasks.rados.rados.0.vpm137.stdout:3471: done (3 left) 2014-08-08T10:49:30.542 INFO:tasks.rados.rados.0.vpm137.stdout:3473: done (2 left) 2014-08-08T10:49:30.542 INFO:tasks.rados.rados.0.vpm137.stdout:3475: write oid 326 current snap is 0 2014-08-08T10:49:30.542 INFO:tasks.rados.rados.0.vpm137.stdout:3475: seq_num 2141 ranges {429207=454762,1523883=525510,2669828=526925,3629206=1} 2014-08-08T10:49:30.571 INFO:tasks.rados.rados.0.vpm137.stdout:3475: writing vpm1375878-326 from 429207 to 883969 tid 1 2014-08-08T10:49:30.611 INFO:tasks.rados.rados.0.vpm137.stdout:3475: writing vpm1375878-326 from 1523883 to 2049393 tid 2 2014-08-08T10:49:30.650 INFO:tasks.rados.rados.0.vpm137.stdout:3475: writing vpm1375878-326 from 2669828 to 3196753 tid 3 2014-08-08T10:49:30.661 INFO:tasks.rados.rados.0.vpm137.stdout:3475: writing vpm1375878-326 from 3629206 to 3629207 tid 4 2014-08-08T10:49:30.664 INFO:tasks.rados.rados.0.vpm137.stdout:3474: finishing write tid 1 to vpm1375878-386 2014-08-08T10:49:30.664 INFO:tasks.rados.rados.0.vpm137.stdout:3474: finishing write tid 2 to vpm1375878-386 2014-08-08T10:49:30.664 INFO:tasks.rados.rados.0.vpm137.stdout:3476: read oid 26 snap -1 2014-08-08T10:49:30.665 INFO:tasks.rados.rados.0.vpm137.stdout:3477: delete oid 266 current snap is 0 2014-08-08T10:49:30.705 INFO:tasks.rados.rados.0.vpm137.stdout:3472: expect (ObjNum 2069 snap 0 seq_num 2069) 2014-08-08T10:49:31.022 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:33.669 INFO:tasks.rados.rados.0.vpm137.stdout:3472: done (4 left) 2014-08-08T10:49:33.670 INFO:tasks.rados.rados.0.vpm137.stdout:3477: done (3 left) 2014-08-08T10:49:33.670 INFO:tasks.rados.rados.0.vpm137.stdout:3478: read oid 273 snap -1 2014-08-08T10:49:33.671 INFO:tasks.rados.rados.0.vpm137.stdout:3479: write oid 342 current snap is 0 2014-08-08T10:49:33.672 INFO:tasks.rados.rados.0.vpm137.stdout:3479: seq_num 2143 ranges {585617=1} 2014-08-08T10:49:33.680 INFO:tasks.rados.rados.0.vpm137.stdout:3479: writing vpm1375878-342 from 585617 to 585618 tid 1 2014-08-08T10:49:33.681 INFO:tasks.rados.rados.0.vpm137.stdout:3474: finishing write tid 3 to vpm1375878-386 2014-08-08T10:49:33.681 INFO:tasks.rados.rados.0.vpm137.stdout:3474: finishing write tid 5 to vpm1375878-386 2014-08-08T10:49:33.682 INFO:tasks.rados.rados.0.vpm137.stdout:3474: finishing write tid 6 to vpm1375878-386 2014-08-08T10:49:33.682 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 386 v 332 (ObjNum 2140 snap 0 seq_num 2140) dirty exists 2014-08-08T10:49:33.682 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 1 to vpm1375878-326 2014-08-08T10:49:33.682 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 2 to vpm1375878-326 2014-08-08T10:49:33.683 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 3 to vpm1375878-326 2014-08-08T10:49:33.683 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 4 to vpm1375878-326 2014-08-08T10:49:33.683 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 6 to vpm1375878-326 2014-08-08T10:49:33.683 INFO:tasks.rados.rados.0.vpm137.stdout:3475: finishing write tid 7 to vpm1375878-326 2014-08-08T10:49:33.684 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 326 v 388 (ObjNum 2141 snap 0 seq_num 2141) dirty exists 2014-08-08T10:49:33.684 INFO:tasks.rados.rados.0.vpm137.stdout:3476: expect (ObjNum 2039 snap 0 seq_num 2039) 2014-08-08T10:49:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3474: done (4 left) 2014-08-08T10:49:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3475: done (3 left) 2014-08-08T10:49:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3476: done (2 left) 2014-08-08T10:49:34.002 INFO:tasks.rados.rados.0.vpm137.stdout:3480: read oid 464 snap -1 2014-08-08T10:49:34.002 INFO:tasks.rados.rados.0.vpm137.stdout:3481: write oid 434 current snap is 0 2014-08-08T10:49:34.003 INFO:tasks.rados.rados.0.vpm137.stdout:3481: seq_num 2144 ranges {529424=406271,1329423=1} 2014-08-08T10:49:34.018 INFO:tasks.rados.rados.0.vpm137.stdout:3481: writing vpm1375878-434 from 529424 to 935695 tid 1 2014-08-08T10:49:34.027 INFO:tasks.rados.rados.0.vpm137.stdout:3481: writing vpm1375878-434 from 1329423 to 1329424 tid 2 2014-08-08T10:49:34.028 INFO:tasks.rados.rados.0.vpm137.stdout:3478: expect (ObjNum 1934 snap 0 seq_num 1934) 2014-08-08T10:49:34.595 INFO:tasks.rados.rados.0.vpm137.stdout:3478: done (3 left) 2014-08-08T10:49:34.595 INFO:tasks.rados.rados.0.vpm137.stdout:3482: read oid 443 snap -1 2014-08-08T10:49:34.595 INFO:tasks.rados.rados.0.vpm137.stdout:3483: read oid 240 snap -1 2014-08-08T10:49:34.595 INFO:tasks.rados.rados.0.vpm137.stdout:3484: read oid 204 snap -1 2014-08-08T10:49:34.596 INFO:tasks.rados.rados.0.vpm137.stdout:3485: write oid 392 current snap is 0 2014-08-08T10:49:34.596 INFO:tasks.rados.rados.0.vpm137.stdout:3485: seq_num 2145 ranges {624654=400000} 2014-08-08T10:49:34.614 INFO:tasks.rados.rados.0.vpm137.stdout:3485: writing vpm1375878-392 from 624654 to 1024654 tid 1 2014-08-08T10:49:34.616 INFO:tasks.rados.rados.0.vpm137.stdout:3479: finishing write tid 1 to vpm1375878-342 2014-08-08T10:49:34.616 INFO:tasks.rados.rados.0.vpm137.stdout:3479: finishing write tid 3 to vpm1375878-342 2014-08-08T10:49:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:3479: finishing write tid 4 to vpm1375878-342 2014-08-08T10:49:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 342 v 416 (ObjNum 2143 snap 0 seq_num 2143) dirty exists 2014-08-08T10:49:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:3481: finishing write tid 1 to vpm1375878-434 2014-08-08T10:49:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:3481: finishing write tid 2 to vpm1375878-434 2014-08-08T10:49:34.618 INFO:tasks.rados.rados.0.vpm137.stdout:3481: finishing write tid 4 to vpm1375878-434 2014-08-08T10:49:34.618 INFO:tasks.rados.rados.0.vpm137.stdout:3479: done (6 left) 2014-08-08T10:49:34.619 INFO:tasks.rados.rados.0.vpm137.stdout:3486: write oid 401 current snap is 0 2014-08-08T10:49:34.619 INFO:tasks.rados.rados.0.vpm137.stdout:3486: seq_num 2146 ranges {685836=750904,1836880=448956} 2014-08-08T10:49:34.656 INFO:tasks.rados.rados.0.vpm137.stdout:3486: writing vpm1375878-401 from 685836 to 1436740 tid 1 2014-08-08T10:49:34.692 INFO:tasks.rados.rados.0.vpm137.stdout:3486: writing vpm1375878-401 from 1836880 to 2285836 tid 2 2014-08-08T10:49:34.692 INFO:tasks.rados.rados.0.vpm137.stdout:3484: expect (ObjNum 2004 snap 0 seq_num 2004) 2014-08-08T10:49:34.711 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:34.737 INFO:tasks.rados.rados.0.vpm137.stdout:3483: done (6 left) 2014-08-08T10:49:34.737 INFO:tasks.rados.rados.0.vpm137.stdout:3484: done (5 left) 2014-08-08T10:49:34.738 INFO:tasks.rados.rados.0.vpm137.stdout:3487: delete oid 66 current snap is 0 2014-08-08T10:49:34.738 INFO:tasks.rados.rados.0.vpm137.stdout:3480: expect (ObjNum 1247 snap 0 seq_num 1247) 2014-08-08T10:49:34.772 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:35.038 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 825999586, u'bytes_used': 1778909184, u'op_per_sec': 16, u'version': 1664, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1540141, u'write_bytes_sec': 1808134, u'bytes_avail': 427508105216}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:35.039 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:35.234 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:35.235 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:35.432 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:36.340 INFO:tasks.rados.rados.0.vpm137.stdout:3480: done (5 left) 2014-08-08T10:49:36.340 INFO:tasks.rados.rados.0.vpm137.stdout:3487: done (4 left) 2014-08-08T10:49:36.341 INFO:tasks.rados.rados.0.vpm137.stdout:3488: read oid 327 snap -1 2014-08-08T10:49:36.341 INFO:tasks.rados.rados.0.vpm137.stdout:3489: write oid 86 current snap is 0 2014-08-08T10:49:36.342 INFO:tasks.rados.rados.0.vpm137.stdout:3489: seq_num 2148 ranges {725895=729585,2063096=554797,2725894=1} 2014-08-08T10:49:36.393 INFO:tasks.rados.rados.0.vpm137.stdout:3489: writing vpm1375878-86 from 725895 to 1455480 tid 1 2014-08-08T10:49:36.442 INFO:tasks.rados.rados.0.vpm137.stdout:3489: writing vpm1375878-86 from 2063096 to 2617893 tid 2 2014-08-08T10:49:36.444 INFO:tasks.rados.rados.0.vpm137.stdout:3489: writing vpm1375878-86 from 2725894 to 2725895 tid 3 2014-08-08T10:49:36.445 INFO:tasks.rados.rados.0.vpm137.stdout:3482: expect (ObjNum 2129 snap 0 seq_num 2129) 2014-08-08T10:49:37.093 INFO:tasks.rados.rados.0.vpm137.stdout:3482: done (5 left) 2014-08-08T10:49:37.094 INFO:tasks.rados.rados.0.vpm137.stdout:3490: delete oid 393 current snap is 0 2014-08-08T10:49:37.096 INFO:tasks.rados.rados.0.vpm137.stdout:3481: finishing write tid 5 to vpm1375878-434 2014-08-08T10:49:37.096 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 434 v 395 (ObjNum 2144 snap 0 seq_num 2144) dirty exists 2014-08-08T10:49:37.096 INFO:tasks.rados.rados.0.vpm137.stdout:3485: finishing write tid 1 to vpm1375878-392 2014-08-08T10:49:37.097 INFO:tasks.rados.rados.0.vpm137.stdout:3485: finishing write tid 3 to vpm1375878-392 2014-08-08T10:49:37.097 INFO:tasks.rados.rados.0.vpm137.stdout:3485: finishing write tid 4 to vpm1375878-392 2014-08-08T10:49:37.097 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 392 v 385 (ObjNum 2145 snap 0 seq_num 2145) dirty exists 2014-08-08T10:49:37.097 INFO:tasks.rados.rados.0.vpm137.stdout:3486: finishing write tid 1 to vpm1375878-401 2014-08-08T10:49:37.098 INFO:tasks.rados.rados.0.vpm137.stdout:3486: finishing write tid 2 to vpm1375878-401 2014-08-08T10:49:37.098 INFO:tasks.rados.rados.0.vpm137.stdout:3486: finishing write tid 4 to vpm1375878-401 2014-08-08T10:49:37.098 INFO:tasks.rados.rados.0.vpm137.stdout:3486: finishing write tid 5 to vpm1375878-401 2014-08-08T10:49:37.098 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 401 v 583 (ObjNum 2146 snap 0 seq_num 2146) dirty exists 2014-08-08T10:49:37.099 INFO:tasks.rados.rados.0.vpm137.stdout:3488: expect (ObjNum 2016 snap 0 seq_num 2016) 2014-08-08T10:49:37.621 INFO:tasks.rados.rados.0.vpm137.stdout:3489: finishing write tid 1 to vpm1375878-86 2014-08-08T10:49:37.621 INFO:tasks.rados.rados.0.vpm137.stdout:3489: finishing write tid 2 to vpm1375878-86 2014-08-08T10:49:37.622 INFO:tasks.rados.rados.0.vpm137.stdout:3489: finishing write tid 3 to vpm1375878-86 2014-08-08T10:49:37.622 INFO:tasks.rados.rados.0.vpm137.stdout:3489: finishing write tid 5 to vpm1375878-86 2014-08-08T10:49:37.622 INFO:tasks.rados.rados.0.vpm137.stdout:3489: finishing write tid 6 to vpm1375878-86 2014-08-08T10:49:37.622 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 86 v 420 (ObjNum 2148 snap 0 seq_num 2148) dirty exists 2014-08-08T10:49:37.669 INFO:tasks.rados.rados.0.vpm137.stdout:3481: done (5 left) 2014-08-08T10:49:37.669 INFO:tasks.rados.rados.0.vpm137.stdout:3485: done (4 left) 2014-08-08T10:49:37.670 INFO:tasks.rados.rados.0.vpm137.stdout:3486: done (3 left) 2014-08-08T10:49:37.670 INFO:tasks.rados.rados.0.vpm137.stdout:3488: done (2 left) 2014-08-08T10:49:37.670 INFO:tasks.rados.rados.0.vpm137.stdout:3489: done (1 left) 2014-08-08T10:49:37.670 INFO:tasks.rados.rados.0.vpm137.stdout:3490: done (0 left) 2014-08-08T10:49:37.671 INFO:tasks.rados.rados.0.vpm137.stdout:3491: read oid 20 snap -1 2014-08-08T10:49:37.671 INFO:tasks.rados.rados.0.vpm137.stdout:3492: write oid 105 current snap is 0 2014-08-08T10:49:37.672 INFO:tasks.rados.rados.0.vpm137.stdout:3492: seq_num 2150 ranges {765954=425641,1856288=695187,3075161=90793} 2014-08-08T10:49:37.689 INFO:tasks.rados.rados.0.vpm137.stdout:3492: writing vpm1375878-105 from 765954 to 1191595 tid 1 2014-08-08T10:49:37.739 INFO:tasks.rados.rados.0.vpm137.stdout:3492: writing vpm1375878-105 from 1856288 to 2551475 tid 2 2014-08-08T10:49:37.760 INFO:tasks.rados.rados.0.vpm137.stdout:3492: writing vpm1375878-105 from 3075161 to 3165954 tid 3 2014-08-08T10:49:37.764 INFO:tasks.rados.rados.0.vpm137.stdout:3491: expect (ObjNum 1726 snap 0 seq_num 1726) 2014-08-08T10:49:38.478 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:39.915 INFO:tasks.rados.rados.0.vpm137.stdout:3491: done (1 left) 2014-08-08T10:49:39.916 INFO:tasks.rados.rados.0.vpm137.stdout:3493: write oid 141 current snap is 0 2014-08-08T10:49:39.916 INFO:tasks.rados.rados.0.vpm137.stdout:3493: seq_num 2151 ranges {427135=1} 2014-08-08T10:49:39.921 INFO:tasks.rados.rados.0.vpm137.stdout:3493: writing vpm1375878-141 from 427135 to 427136 tid 1 2014-08-08T10:49:39.922 INFO:tasks.rados.rados.0.vpm137.stdout:3492: finishing write tid 1 to vpm1375878-105 2014-08-08T10:49:39.922 INFO:tasks.rados.rados.0.vpm137.stdout:3492: finishing write tid 2 to vpm1375878-105 2014-08-08T10:49:39.922 INFO:tasks.rados.rados.0.vpm137.stdout:3492: finishing write tid 3 to vpm1375878-105 2014-08-08T10:49:39.923 INFO:tasks.rados.rados.0.vpm137.stdout:3492: finishing write tid 5 to vpm1375878-105 2014-08-08T10:49:39.923 INFO:tasks.rados.rados.0.vpm137.stdout:3492: finishing write tid 6 to vpm1375878-105 2014-08-08T10:49:39.923 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 105 v 321 (ObjNum 2150 snap 0 seq_num 2150) dirty exists 2014-08-08T10:49:39.923 INFO:tasks.rados.rados.0.vpm137.stdout:3492: done (1 left) 2014-08-08T10:49:39.924 INFO:tasks.rados.rados.0.vpm137.stdout:3494: read oid 120 snap -1 2014-08-08T10:49:39.924 INFO:tasks.rados.rados.0.vpm137.stdout:3495: read oid 208 snap -1 2014-08-08T10:49:39.925 INFO:tasks.rados.rados.0.vpm137.stdout:3496: write oid 291 current snap is 0 2014-08-08T10:49:39.925 INFO:tasks.rados.rados.0.vpm137.stdout:3496: seq_num 2152 ranges {771966=400000} 2014-08-08T10:49:39.955 INFO:tasks.rados.rados.0.vpm137.stdout:3496: writing vpm1375878-291 from 771966 to 1171966 tid 1 2014-08-08T10:49:39.961 INFO:tasks.rados.rados.0.vpm137.stdout:3497: write oid 363 current snap is 0 2014-08-08T10:49:39.961 INFO:tasks.rados.rados.0.vpm137.stdout:3497: seq_num 2153 ranges {466172=400000} 2014-08-08T10:49:39.992 INFO:tasks.rados.rados.0.vpm137.stdout:3497: writing vpm1375878-363 from 466172 to 866172 tid 1 2014-08-08T10:49:39.993 INFO:tasks.rados.rados.0.vpm137.stdout:3493: finishing write tid 1 to vpm1375878-141 2014-08-08T10:49:39.993 INFO:tasks.rados.rados.0.vpm137.stdout:3498: write oid 481 current snap is 0 2014-08-08T10:49:39.993 INFO:tasks.rados.rados.0.vpm137.stdout:3498: seq_num 2154 ranges {411001=500379,1325297=285704} 2014-08-08T10:49:40.026 INFO:tasks.rados.rados.0.vpm137.stdout:3498: writing vpm1375878-481 from 411001 to 911380 tid 1 2014-08-08T10:49:40.062 INFO:tasks.rados.rados.0.vpm137.stdout:3498: writing vpm1375878-481 from 1325297 to 1611001 tid 2 2014-08-08T10:49:40.063 INFO:tasks.rados.rados.0.vpm137.stdout:3493: finishing write tid 3 to vpm1375878-141 2014-08-08T10:49:40.064 INFO:tasks.rados.rados.0.vpm137.stdout:3493: finishing write tid 4 to vpm1375878-141 2014-08-08T10:49:40.064 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 141 v 370 (ObjNum 2151 snap 0 seq_num 2151) dirty exists 2014-08-08T10:49:40.064 INFO:tasks.rados.rados.0.vpm137.stdout:3496: finishing write tid 1 to vpm1375878-291 2014-08-08T10:49:40.064 INFO:tasks.rados.rados.0.vpm137.stdout:3496: finishing write tid 3 to vpm1375878-291 2014-08-08T10:49:40.065 INFO:tasks.rados.rados.0.vpm137.stdout:3494: expect (ObjNum 2130 snap 0 seq_num 2130) 2014-08-08T10:49:41.223 INFO:tasks.rados.rados.0.vpm137.stdout:3493: done (5 left) 2014-08-08T10:49:41.224 INFO:tasks.rados.rados.0.vpm137.stdout:3494: done (4 left) 2014-08-08T10:49:41.224 INFO:tasks.rados.rados.0.vpm137.stdout:3499: write oid 65 current snap is 0 2014-08-08T10:49:41.225 INFO:tasks.rados.rados.0.vpm137.stdout:3499: seq_num 2155 ranges {471159=507255,1403333=431635,2577416=293743} 2014-08-08T10:49:41.253 INFO:tasks.rados.rados.0.vpm137.stdout:3499: writing vpm1375878-65 from 471159 to 978414 tid 1 2014-08-08T10:49:41.282 INFO:tasks.rados.rados.0.vpm137.stdout:3499: writing vpm1375878-65 from 1403333 to 1834968 tid 2 2014-08-08T10:49:41.301 INFO:tasks.rados.rados.0.vpm137.stdout:3499: writing vpm1375878-65 from 2577416 to 2871159 tid 3 2014-08-08T10:49:41.302 INFO:tasks.rados.rados.0.vpm137.stdout:3496: finishing write tid 4 to vpm1375878-291 2014-08-08T10:49:41.303 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 291 v 323 (ObjNum 2152 snap 0 seq_num 2152) dirty exists 2014-08-08T10:49:41.304 INFO:tasks.rados.rados.0.vpm137.stdout:3497: finishing write tid 1 to vpm1375878-363 2014-08-08T10:49:41.304 INFO:tasks.rados.rados.0.vpm137.stdout:3497: finishing write tid 3 to vpm1375878-363 2014-08-08T10:49:41.304 INFO:tasks.rados.rados.0.vpm137.stdout:3497: finishing write tid 4 to vpm1375878-363 2014-08-08T10:49:41.304 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 363 v 390 (ObjNum 2153 snap 0 seq_num 2153) dirty exists 2014-08-08T10:49:41.305 INFO:tasks.rados.rados.0.vpm137.stdout:3495: expect (ObjNum 1789 snap 0 seq_num 1789) 2014-08-08T10:49:41.669 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:41.714 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:41.942 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 822037262, u'bytes_used': 1768206336, u'op_per_sec': 12, u'version': 1666, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2153170, u'write_bytes_sec': 879241, u'bytes_avail': 427518808064}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:41.943 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:42.135 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:42.136 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:42.325 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:42.719 INFO:tasks.rados.rados.0.vpm137.stdout:3495: done (4 left) 2014-08-08T10:49:42.719 INFO:tasks.rados.rados.0.vpm137.stdout:3496: done (3 left) 2014-08-08T10:49:42.720 INFO:tasks.rados.rados.0.vpm137.stdout:3497: done (2 left) 2014-08-08T10:49:42.720 INFO:tasks.rados.rados.0.vpm137.stdout:3500: read oid 283 snap -1 2014-08-08T10:49:42.720 INFO:tasks.rados.rados.0.vpm137.stdout:3501: read oid 398 snap -1 2014-08-08T10:49:42.721 INFO:tasks.rados.rados.0.vpm137.stdout:3502: read oid 73 snap -1 2014-08-08T10:49:42.721 INFO:tasks.rados.rados.0.vpm137.stdout:3503: write oid 241 current snap is 0 2014-08-08T10:49:42.721 INFO:tasks.rados.rados.0.vpm137.stdout:3503: seq_num 2156 ranges {567413=727636,1765018=501830,2567412=1} 2014-08-08T10:49:42.759 INFO:tasks.rados.rados.0.vpm137.stdout:3503: writing vpm1375878-241 from 567413 to 1295049 tid 1 2014-08-08T10:49:42.782 INFO:tasks.rados.rados.0.vpm137.stdout:3503: writing vpm1375878-241 from 1765018 to 2266848 tid 2 2014-08-08T10:49:42.787 INFO:tasks.rados.rados.0.vpm137.stdout:3503: writing vpm1375878-241 from 2567412 to 2567413 tid 3 2014-08-08T10:49:42.788 INFO:tasks.rados.rados.0.vpm137.stdout:3498: finishing write tid 1 to vpm1375878-481 2014-08-08T10:49:42.788 INFO:tasks.rados.rados.0.vpm137.stdout:3498: finishing write tid 2 to vpm1375878-481 2014-08-08T10:49:42.788 INFO:tasks.rados.rados.0.vpm137.stdout:3498: finishing write tid 4 to vpm1375878-481 2014-08-08T10:49:42.788 INFO:tasks.rados.rados.0.vpm137.stdout:3498: finishing write tid 5 to vpm1375878-481 2014-08-08T10:49:42.788 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 481 v 487 (ObjNum 2154 snap 0 seq_num 2154) dirty exists 2014-08-08T10:49:42.789 INFO:tasks.rados.rados.0.vpm137.stdout:3500: expect (ObjNum 1065 snap 0 seq_num 1065) 2014-08-08T10:49:43.197 INFO:tasks.rados.rados.0.vpm137.stdout:3498: done (5 left) 2014-08-08T10:49:43.197 INFO:tasks.rados.rados.0.vpm137.stdout:3500: done (4 left) 2014-08-08T10:49:43.197 INFO:tasks.rados.rados.0.vpm137.stdout:3504: read oid 40 snap -1 2014-08-08T10:49:43.197 INFO:tasks.rados.rados.0.vpm137.stdout:3505: read oid 438 snap -1 2014-08-08T10:49:43.197 INFO:tasks.rados.rados.0.vpm137.stdout:3506: read oid 108 snap -1 2014-08-08T10:49:43.198 INFO:tasks.rados.rados.0.vpm137.stdout:3507: read oid 157 snap -1 2014-08-08T10:49:43.198 INFO:tasks.rados.rados.0.vpm137.stdout:3508: write oid 463 current snap is 0 2014-08-08T10:49:43.198 INFO:tasks.rados.rados.0.vpm137.stdout:3508: seq_num 2157 ranges {512243=451888,1480174=723449,2863515=448728} 2014-08-08T10:49:43.224 INFO:tasks.rados.rados.0.vpm137.stdout:3508: writing vpm1375878-463 from 512243 to 964131 tid 1 2014-08-08T10:49:43.259 INFO:tasks.rados.rados.0.vpm137.stdout:3508: writing vpm1375878-463 from 1480174 to 2203623 tid 2 2014-08-08T10:49:43.280 INFO:tasks.rados.rados.0.vpm137.stdout:3508: writing vpm1375878-463 from 2863515 to 3312243 tid 3 2014-08-08T10:49:43.283 INFO:tasks.rados.rados.0.vpm137.stdout:3509: write oid 286 current snap is 0 2014-08-08T10:49:43.283 INFO:tasks.rados.rados.0.vpm137.stdout:3509: seq_num 2158 ranges {56047=1} 2014-08-08T10:49:43.284 INFO:tasks.rados.rados.0.vpm137.stdout:3509: writing vpm1375878-286 from 56047 to 56048 tid 1 2014-08-08T10:49:43.285 INFO:tasks.rados.rados.0.vpm137.stdout:3510: write oid 104 current snap is 0 2014-08-08T10:49:43.285 INFO:tasks.rados.rados.0.vpm137.stdout:3510: seq_num 2159 ranges {552302=430569,1555990=463840,2446766=692802,3752301=1} 2014-08-08T10:49:43.302 INFO:tasks.rados.rados.0.vpm137.stdout:3510: writing vpm1375878-104 from 552302 to 982871 tid 1 2014-08-08T10:49:43.322 INFO:tasks.rados.rados.0.vpm137.stdout:3510: writing vpm1375878-104 from 1555990 to 2019830 tid 2 2014-08-08T10:49:43.349 INFO:tasks.rados.rados.0.vpm137.stdout:3510: writing vpm1375878-104 from 2446766 to 3139568 tid 3 2014-08-08T10:49:43.356 INFO:tasks.rados.rados.0.vpm137.stdout:3510: writing vpm1375878-104 from 3752301 to 3752302 tid 4 2014-08-08T10:49:43.357 INFO:tasks.rados.rados.0.vpm137.stdout:3511: read oid 261 snap -1 2014-08-08T10:49:43.358 INFO:tasks.rados.rados.0.vpm137.stdout:3512: read oid 105 snap -1 2014-08-08T10:49:43.358 INFO:tasks.rados.rados.0.vpm137.stdout:3513: read oid 355 snap -1 2014-08-08T10:49:43.359 INFO:tasks.rados.rados.0.vpm137.stdout:3514: write oid 425 current snap is 0 2014-08-08T10:49:43.359 INFO:tasks.rados.rados.0.vpm137.stdout:3514: seq_num 2160 ranges {612460=400000} 2014-08-08T10:49:43.376 INFO:tasks.rados.rados.0.vpm137.stdout:3514: writing vpm1375878-425 from 612460 to 1012460 tid 1 2014-08-08T10:49:43.379 INFO:tasks.rados.rados.0.vpm137.stdout:3515: write oid 52 current snap is 0 2014-08-08T10:49:43.379 INFO:tasks.rados.rados.0.vpm137.stdout:3515: seq_num 2161 ranges {556266=526625,1713086=43180} 2014-08-08T10:49:43.403 INFO:tasks.rados.rados.0.vpm137.stdout:3515: writing vpm1375878-52 from 556266 to 1082891 tid 1 2014-08-08T10:49:43.412 INFO:tasks.rados.rados.0.vpm137.stdout:3515: writing vpm1375878-52 from 1713086 to 1756266 tid 2 2014-08-08T10:49:43.413 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:49:44.525 INFO:tasks.rados.rados.0.vpm137.stdout:3502: expect (ObjNum 754 snap 0 seq_num 754) 2014-08-08T10:49:45.271 INFO:tasks.rados.rados.0.vpm137.stdout:3502: done (15 left) 2014-08-08T10:49:45.271 INFO:tasks.rados.rados.0.vpm137.stdout:3516: write oid 475 current snap is 0 2014-08-08T10:49:45.272 INFO:tasks.rados.rados.0.vpm137.stdout:3516: seq_num 2162 ranges {652519=498430,1452518=1} 2014-08-08T10:49:45.292 INFO:tasks.rados.rados.0.vpm137.stdout:3516: writing vpm1375878-475 from 652519 to 1150949 tid 1 2014-08-08T10:49:45.299 INFO:tasks.rados.rados.0.vpm137.stdout:3516: writing vpm1375878-475 from 1452518 to 1452519 tid 2 2014-08-08T10:49:45.299 INFO:tasks.rados.rados.0.vpm137.stdout:3499: finishing write tid 1 to vpm1375878-65 2014-08-08T10:49:45.300 INFO:tasks.rados.rados.0.vpm137.stdout:3499: finishing write tid 2 to vpm1375878-65 2014-08-08T10:49:45.300 INFO:tasks.rados.rados.0.vpm137.stdout:3501: expect (ObjNum 1559 snap 0 seq_num 1559) 2014-08-08T10:49:45.357 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:45.553 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:45.603 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:45.866 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 825030548, u'bytes_used': 1765650432, u'op_per_sec': 13, u'version': 1667, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2440224, u'write_bytes_sec': 1073739, u'bytes_avail': 427521363968}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:45.867 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:45.893 INFO:tasks.rados.rados.0.vpm137.stdout:3501: done (15 left) 2014-08-08T10:49:45.893 INFO:tasks.rados.rados.0.vpm137.stdout:3517: write oid 380 current snap is 0 2014-08-08T10:49:45.894 INFO:tasks.rados.rados.0.vpm137.stdout:3517: seq_num 2163 ranges {596325=505306,1540326=655999} 2014-08-08T10:49:45.925 INFO:tasks.rados.rados.0.vpm137.stdout:3517: writing vpm1375878-380 from 596325 to 1101631 tid 1 2014-08-08T10:49:45.962 INFO:tasks.rados.rados.0.vpm137.stdout:3517: writing vpm1375878-380 from 1540326 to 2196325 tid 2 2014-08-08T10:49:45.969 INFO:tasks.rados.rados.0.vpm137.stdout:3499: finishing write tid 3 to vpm1375878-65 2014-08-08T10:49:45.969 INFO:tasks.rados.rados.0.vpm137.stdout:3503: finishing write tid 1 to vpm1375878-241 2014-08-08T10:49:45.970 INFO:tasks.rados.rados.0.vpm137.stdout:3499: finishing write tid 5 to vpm1375878-65 2014-08-08T10:49:45.970 INFO:tasks.rados.rados.0.vpm137.stdout:3499: finishing write tid 6 to vpm1375878-65 2014-08-08T10:49:45.970 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 65 v 394 (ObjNum 2155 snap 0 seq_num 2155) dirty exists 2014-08-08T10:49:45.970 INFO:tasks.rados.rados.0.vpm137.stdout:3511: expect (ObjNum 1010 snap 0 seq_num 1010) 2014-08-08T10:49:46.121 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:46.122 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:46.341 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:46.711 INFO:tasks.rados.rados.0.vpm137.stdout:3499: done (15 left) 2014-08-08T10:49:46.711 INFO:tasks.rados.rados.0.vpm137.stdout:3506: done (14 left) 2014-08-08T10:49:46.712 INFO:tasks.rados.rados.0.vpm137.stdout:3507: done (13 left) 2014-08-08T10:49:46.712 INFO:tasks.rados.rados.0.vpm137.stdout:3511: done (12 left) 2014-08-08T10:49:46.713 INFO:tasks.rados.rados.0.vpm137.stdout:3518: write oid 399 current snap is 0 2014-08-08T10:49:46.713 INFO:tasks.rados.rados.0.vpm137.stdout:3518: seq_num 2164 ranges {657507=478134,1867962=601312,3114973=342534} 2014-08-08T10:49:46.742 INFO:tasks.rados.rados.0.vpm137.stdout:3518: writing vpm1375878-399 from 657507 to 1135641 tid 1 2014-08-08T10:49:46.788 INFO:tasks.rados.rados.0.vpm137.stdout:3518: writing vpm1375878-399 from 1867962 to 2469274 tid 2 2014-08-08T10:49:46.816 INFO:tasks.rados.rados.0.vpm137.stdout:3518: writing vpm1375878-399 from 3114973 to 3457507 tid 3 2014-08-08T10:49:46.818 INFO:tasks.rados.rados.0.vpm137.stdout:3503: finishing write tid 2 to vpm1375878-241 2014-08-08T10:49:46.818 INFO:tasks.rados.rados.0.vpm137.stdout:3503: finishing write tid 3 to vpm1375878-241 2014-08-08T10:49:46.818 INFO:tasks.rados.rados.0.vpm137.stdout:3503: finishing write tid 5 to vpm1375878-241 2014-08-08T10:49:46.819 INFO:tasks.rados.rados.0.vpm137.stdout:3508: finishing write tid 1 to vpm1375878-463 2014-08-08T10:49:46.819 INFO:tasks.rados.rados.0.vpm137.stdout:3508: finishing write tid 2 to vpm1375878-463 2014-08-08T10:49:46.819 INFO:tasks.rados.rados.0.vpm137.stdout:3508: finishing write tid 3 to vpm1375878-463 2014-08-08T10:49:46.819 INFO:tasks.rados.rados.0.vpm137.stdout:3508: finishing write tid 5 to vpm1375878-463 2014-08-08T10:49:46.820 INFO:tasks.rados.rados.0.vpm137.stdout:3512: expect (ObjNum 2150 snap 0 seq_num 2150) 2014-08-08T10:49:48.671 INFO:tasks.rados.rados.0.vpm137.stdout:3512: done (12 left) 2014-08-08T10:49:49.845 INFO:tasks.rados.rados.0.vpm137.stdout:3519: write oid 216 current snap is 0 2014-08-08T10:49:49.845 INFO:tasks.rados.rados.0.vpm137.stdout:3519: seq_num 2165 ranges {752737=601363,1848847=422931,2818023=334714} 2014-08-08T10:49:49.847 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3519: writing vpm1375878-216 from 752737 to 1354100 tid 1 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3519: writing vpm1375878-216 from 1848847 to 2271778 tid 2 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3519: writing vpm1375878-216 from 2818023 to 3152737 tid 3 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3509: finishing write tid 1 to vpm1375878-286 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3509: finishing write tid 3 to vpm1375878-286 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3515: finishing write tid 1 to vpm1375878-52 2014-08-08T10:49:49.848 INFO:tasks.rados.rados.0.vpm137.stdout:3515: finishing write tid 2 to vpm1375878-52 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3515: finishing write tid 4 to vpm1375878-52 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3505: expect (ObjNum 1485 snap 0 seq_num 1485) 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3505: done (12 left) 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3520: write oid 47 current snap is 0 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3520: seq_num 2166 ranges {697566=573167,1811555=741702,3117424=738096,3897565=1} 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3520: writing vpm1375878-47 from 697566 to 1270733 tid 1 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3520: writing vpm1375878-47 from 1811555 to 2553257 tid 2 2014-08-08T10:49:49.849 INFO:tasks.rados.rados.0.vpm137.stdout:3520: writing vpm1375878-47 from 3117424 to 3855520 tid 3 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:3520: writing vpm1375878-47 from 3897565 to 3897566 tid 4 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:3503: finishing write tid 6 to vpm1375878-241 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 241 v 398 (ObjNum 2156 snap 0 seq_num 2156) dirty exists 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:3508: finishing write tid 6 to vpm1375878-463 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 463 v 337 (ObjNum 2157 snap 0 seq_num 2157) dirty exists 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:3515: finishing write tid 5 to vpm1375878-52 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 52 v 340 (ObjNum 2161 snap 0 seq_num 2161) dirty exists 2014-08-08T10:49:49.850 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 1 to vpm1375878-104 2014-08-08T10:49:49.851 INFO:tasks.rados.rados.0.vpm137.stdout:3504: expect (ObjNum 1893 snap 0 seq_num 1893) 2014-08-08T10:49:50.077 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:50.122 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:50.301 INFO:tasks.rados.rados.0.vpm137.stdout:3503: done (12 left) 2014-08-08T10:49:50.301 INFO:tasks.rados.rados.0.vpm137.stdout:3504: done (11 left) 2014-08-08T10:49:50.302 INFO:tasks.rados.rados.0.vpm137.stdout:3508: done (10 left) 2014-08-08T10:49:50.302 INFO:tasks.rados.rados.0.vpm137.stdout:3513: done (9 left) 2014-08-08T10:49:50.302 INFO:tasks.rados.rados.0.vpm137.stdout:3515: done (8 left) 2014-08-08T10:49:50.303 INFO:tasks.rados.rados.0.vpm137.stdout:3521: read oid 500 snap -1 2014-08-08T10:49:50.303 INFO:tasks.rados.rados.0.vpm137.stdout:3522: read oid 259 snap -1 2014-08-08T10:49:50.304 INFO:tasks.rados.rados.0.vpm137.stdout:3523: write oid 387 current snap is 0 2014-08-08T10:49:50.304 INFO:tasks.rados.rados.0.vpm137.stdout:3523: seq_num 2167 ranges {642395=1} 2014-08-08T10:49:50.311 INFO:tasks.rados.rados.0.vpm137.stdout:3523: writing vpm1375878-387 from 642395 to 642396 tid 1 2014-08-08T10:49:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3509: finishing write tid 4 to vpm1375878-286 2014-08-08T10:49:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 286 v 325 (ObjNum 2158 snap 0 seq_num 2158) dirty exists 2014-08-08T10:49:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 2 to vpm1375878-104 2014-08-08T10:49:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 3 to vpm1375878-104 2014-08-08T10:49:50.314 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 4 to vpm1375878-104 2014-08-08T10:49:50.314 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 6 to vpm1375878-104 2014-08-08T10:49:50.314 INFO:tasks.rados.rados.0.vpm137.stdout:3514: finishing write tid 1 to vpm1375878-425 2014-08-08T10:49:50.315 INFO:tasks.rados.rados.0.vpm137.stdout:3514: finishing write tid 3 to vpm1375878-425 2014-08-08T10:49:50.315 INFO:tasks.rados.rados.0.vpm137.stdout:3510: finishing write tid 7 to vpm1375878-104 2014-08-08T10:49:50.315 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 104 v 338 (ObjNum 2159 snap 0 seq_num 2159) dirty exists 2014-08-08T10:49:50.315 INFO:tasks.rados.rados.0.vpm137.stdout:3514: finishing write tid 4 to vpm1375878-425 2014-08-08T10:49:50.316 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 425 v 340 (ObjNum 2160 snap 0 seq_num 2160) dirty exists 2014-08-08T10:49:50.316 INFO:tasks.rados.rados.0.vpm137.stdout:3516: finishing write tid 1 to vpm1375878-475 2014-08-08T10:49:50.316 INFO:tasks.rados.rados.0.vpm137.stdout:3516: finishing write tid 2 to vpm1375878-475 2014-08-08T10:49:50.316 INFO:tasks.rados.rados.0.vpm137.stdout:3516: finishing write tid 4 to vpm1375878-475 2014-08-08T10:49:50.316 INFO:tasks.rados.rados.0.vpm137.stdout:3516: finishing write tid 5 to vpm1375878-475 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 475 v 423 (ObjNum 2162 snap 0 seq_num 2162) dirty exists 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3517: finishing write tid 1 to vpm1375878-380 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3517: finishing write tid 2 to vpm1375878-380 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3517: finishing write tid 4 to vpm1375878-380 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3517: finishing write tid 5 to vpm1375878-380 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 380 v 397 (ObjNum 2163 snap 0 seq_num 2163) dirty exists 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3518: finishing write tid 1 to vpm1375878-399 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3518: finishing write tid 2 to vpm1375878-399 2014-08-08T10:49:50.317 INFO:tasks.rados.rados.0.vpm137.stdout:3518: finishing write tid 3 to vpm1375878-399 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3518: finishing write tid 5 to vpm1375878-399 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3518: finishing write tid 6 to vpm1375878-399 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 399 v 402 (ObjNum 2164 snap 0 seq_num 2164) dirty exists 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3519: finishing write tid 1 to vpm1375878-216 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3519: finishing write tid 2 to vpm1375878-216 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3519: finishing write tid 3 to vpm1375878-216 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3519: finishing write tid 5 to vpm1375878-216 2014-08-08T10:49:50.318 INFO:tasks.rados.rados.0.vpm137.stdout:3519: finishing write tid 6 to vpm1375878-216 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 216 v 406 (ObjNum 2165 snap 0 seq_num 2165) dirty exists 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 1 to vpm1375878-47 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 2 to vpm1375878-47 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 3 to vpm1375878-47 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 4 to vpm1375878-47 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 6 to vpm1375878-47 2014-08-08T10:49:50.319 INFO:tasks.rados.rados.0.vpm137.stdout:3520: finishing write tid 7 to vpm1375878-47 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 47 v 330 (ObjNum 2166 snap 0 seq_num 2166) dirty exists 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:3509: done (10 left) 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:3510: done (9 left) 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:3514: done (8 left) 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:3516: done (7 left) 2014-08-08T10:49:50.320 INFO:tasks.rados.rados.0.vpm137.stdout:3517: done (6 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3518: done (5 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3519: done (4 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3520: done (3 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3521: done (2 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3522: done (1 left) 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3524: write oid 163 current snap is 0 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3524: seq_num 2168 ranges {337625=1} 2014-08-08T10:49:50.321 INFO:tasks.rados.rados.0.vpm137.stdout:3524: writing vpm1375878-163 from 337625 to 337626 tid 1 2014-08-08T10:49:50.322 INFO:tasks.rados.rados.0.vpm137.stdout:3525: write oid 384 current snap is 0 2014-08-08T10:49:50.322 INFO:tasks.rados.rados.0.vpm137.stdout:3525: seq_num 2169 ranges {797784=524676,1597783=1} 2014-08-08T10:49:50.355 INFO:tasks.rados.rados.0.vpm137.stdout:3525: writing vpm1375878-384 from 797784 to 1322460 tid 1 2014-08-08T10:49:50.361 INFO:tasks.rados.rados.0.vpm137.stdout:3525: writing vpm1375878-384 from 1597783 to 1597784 tid 2 2014-08-08T10:49:50.362 INFO:tasks.rados.rados.0.vpm137.stdout:3523: finishing write tid 1 to vpm1375878-387 2014-08-08T10:49:50.362 INFO:tasks.rados.rados.0.vpm137.stdout:3523: finishing write tid 3 to vpm1375878-387 2014-08-08T10:49:50.362 INFO:tasks.rados.rados.0.vpm137.stdout:3526: read oid 102 snap -1 2014-08-08T10:49:50.363 INFO:tasks.rados.rados.0.vpm137.stdout:3527: write oid 111 current snap is 0 2014-08-08T10:49:50.363 INFO:tasks.rados.rados.0.vpm137.stdout:3527: seq_num 2170 ranges {493013=497504,1293012=1} 2014-08-08T10:49:50.394 INFO:tasks.rados.rados.0.vpm137.stdout:3527: writing vpm1375878-111 from 493013 to 990517 tid 1 2014-08-08T10:49:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3527: writing vpm1375878-111 from 1293012 to 1293013 tid 2 2014-08-08T10:49:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3524: finishing write tid 1 to vpm1375878-163 2014-08-08T10:49:50.402 INFO:tasks.rados.rados.0.vpm137.stdout:3523: finishing write tid 4 to vpm1375878-387 2014-08-08T10:49:50.402 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 387 v 489 (ObjNum 2167 snap 0 seq_num 2167) dirty exists 2014-08-08T10:49:50.402 INFO:tasks.rados.rados.0.vpm137.stdout:3524: finishing write tid 3 to vpm1375878-163 2014-08-08T10:49:50.402 INFO:tasks.rados.rados.0.vpm137.stdout:3524: finishing write tid 4 to vpm1375878-163 2014-08-08T10:49:50.402 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 163 v 585 (ObjNum 2168 snap 0 seq_num 2168) dirty exists 2014-08-08T10:49:50.403 INFO:tasks.rados.rados.0.vpm137.stdout:3523: done (4 left) 2014-08-08T10:49:50.403 INFO:tasks.rados.rados.0.vpm137.stdout:3524: done (3 left) 2014-08-08T10:49:50.403 INFO:tasks.rados.rados.0.vpm137.stdout:3528: read oid 451 snap -1 2014-08-08T10:49:50.404 INFO:tasks.rados.rados.0.vpm137.stdout:3529: read oid 146 snap -1 2014-08-08T10:49:50.404 INFO:tasks.rados.rados.0.vpm137.stdout:3530: write oid 286 current snap is 0 2014-08-08T10:49:50.404 INFO:tasks.rados.rados.0.vpm137.stdout:3530: seq_num 2171 ranges {437843=505405,1492872=479174,2037842=1} 2014-08-08T10:49:50.438 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 825030548, u'bytes_used': 1765650432, u'op_per_sec': 13, u'version': 1667, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2440224, u'write_bytes_sec': 1073739, u'bytes_avail': 427521363968}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:50.438 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:50.439 INFO:tasks.rados.rados.0.vpm137.stdout:3530: writing vpm1375878-286 from 437843 to 943248 tid 1 2014-08-08T10:49:50.470 INFO:tasks.rados.rados.0.vpm137.stdout:3530: writing vpm1375878-286 from 1492872 to 1972046 tid 2 2014-08-08T10:49:50.471 INFO:tasks.rados.rados.0.vpm137.stdout:3530: writing vpm1375878-286 from 2037842 to 2037843 tid 3 2014-08-08T10:49:50.472 INFO:tasks.rados.rados.0.vpm137.stdout:3525: finishing write tid 1 to vpm1375878-384 2014-08-08T10:49:50.472 INFO:tasks.rados.rados.0.vpm137.stdout:3531: write oid 258 current snap is 0 2014-08-08T10:49:50.472 INFO:tasks.rados.rados.0.vpm137.stdout:3531: seq_num 2172 ranges {782673=477209,1854556=700793,2782672=1} 2014-08-08T10:49:50.506 INFO:tasks.rados.rados.0.vpm137.stdout:3531: writing vpm1375878-258 from 782673 to 1259882 tid 1 2014-08-08T10:49:50.552 INFO:tasks.rados.rados.0.vpm137.stdout:3531: writing vpm1375878-258 from 1854556 to 2555349 tid 2 2014-08-08T10:49:50.555 INFO:tasks.rados.rados.0.vpm137.stdout:3531: writing vpm1375878-258 from 2782672 to 2782673 tid 3 2014-08-08T10:49:50.556 INFO:tasks.rados.rados.0.vpm137.stdout:3525: finishing write tid 2 to vpm1375878-384 2014-08-08T10:49:50.556 INFO:tasks.rados.rados.0.vpm137.stdout:3525: finishing write tid 4 to vpm1375878-384 2014-08-08T10:49:50.556 INFO:tasks.rados.rados.0.vpm137.stdout:3525: finishing write tid 5 to vpm1375878-384 2014-08-08T10:49:50.556 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 384 v 409 (ObjNum 2169 snap 0 seq_num 2169) dirty exists 2014-08-08T10:49:50.557 INFO:tasks.rados.rados.0.vpm137.stdout:3525: done (6 left) 2014-08-08T10:49:50.557 INFO:tasks.rados.rados.0.vpm137.stdout:3532: write oid 466 current snap is 0 2014-08-08T10:49:50.557 INFO:tasks.rados.rados.0.vpm137.stdout:3532: seq_num 2173 ranges {594254=600437,1800367=769964,2994253=1} 2014-08-08T10:49:50.603 INFO:tasks.rados.rados.0.vpm137.stdout:3532: writing vpm1375878-466 from 594254 to 1194691 tid 1 2014-08-08T10:49:50.653 INFO:tasks.rados.rados.0.vpm137.stdout:3532: writing vpm1375878-466 from 1800367 to 2570331 tid 2 2014-08-08T10:49:50.658 INFO:tasks.rados.rados.0.vpm137.stdout:3532: writing vpm1375878-466 from 2994253 to 2994254 tid 3 2014-08-08T10:49:50.659 INFO:tasks.rados.rados.0.vpm137.stdout:3533: write oid 317 current snap is 0 2014-08-08T10:49:50.659 INFO:tasks.rados.rados.0.vpm137.stdout:3533: seq_num 2174 ranges {539084=573266,1764100=440160,2654779=551780,3630551=108533} 2014-08-08T10:49:50.696 INFO:tasks.rados.rados.0.vpm137.stdout:3533: writing vpm1375878-317 from 539084 to 1112350 tid 1 2014-08-08T10:49:50.724 INFO:tasks.rados.rados.0.vpm137.stdout:3533: writing vpm1375878-317 from 1764100 to 2204260 tid 2 2014-08-08T10:49:50.751 INFO:tasks.rados.rados.0.vpm137.stdout:3533: writing vpm1375878-317 from 2654779 to 3206559 tid 3 2014-08-08T10:49:50.758 INFO:tasks.rados.rados.0.vpm137.stdout:3533: writing vpm1375878-317 from 3630551 to 3739084 tid 4 2014-08-08T10:49:50.759 INFO:tasks.rados.rados.0.vpm137.stdout:3534: write oid 408 current snap is 0 2014-08-08T10:49:50.759 INFO:tasks.rados.rados.0.vpm137.stdout:3534: seq_num 2175 ranges {482889=1} 2014-08-08T10:49:50.763 INFO:tasks.rados.rados.0.vpm137.stdout:3534: writing vpm1375878-408 from 482889 to 482890 tid 1 2014-08-08T10:49:50.764 INFO:tasks.rados.rados.0.vpm137.stdout:3535: write oid 129 current snap is 0 2014-08-08T10:49:50.764 INFO:tasks.rados.rados.0.vpm137.stdout:3535: seq_num 2176 ranges {179143=1} 2014-08-08T10:49:50.765 INFO:tasks.rados.rados.0.vpm137.stdout:3535: writing vpm1375878-129 from 179143 to 179144 tid 1 2014-08-08T10:49:50.766 INFO:tasks.rados.rados.0.vpm137.stdout:3536: read oid 134 snap -1 2014-08-08T10:49:50.766 INFO:tasks.rados.rados.0.vpm137.stdout:3537: delete oid 148 current snap is 0 2014-08-08T10:49:50.881 INFO:tasks.rados.rados.0.vpm137.stdout:3527: finishing write tid 1 to vpm1375878-111 2014-08-08T10:49:50.887 INFO:tasks.rados.rados.0.vpm137.stdout:3529: expect (ObjNum 1260 snap 0 seq_num 1260) 2014-08-08T10:49:50.933 INFO:tasks.rados.rados.0.vpm137.stdout:3527: finishing write tid 2 to vpm1375878-111 2014-08-08T10:49:50.935 INFO:tasks.rados.rados.0.vpm137.stdout:3527: finishing write tid 4 to vpm1375878-111 2014-08-08T10:49:50.936 INFO:tasks.rados.rados.0.vpm137.stdout:3527: finishing write tid 5 to vpm1375878-111 2014-08-08T10:49:50.936 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 111 v 588 (ObjNum 2170 snap 0 seq_num 2170) dirty exists 2014-08-08T10:49:50.937 INFO:tasks.rados.rados.0.vpm137.stdout:3530: finishing write tid 1 to vpm1375878-286 2014-08-08T10:49:50.943 INFO:tasks.rados.rados.0.vpm137.stdout:3530: finishing write tid 2 to vpm1375878-286 2014-08-08T10:49:50.960 INFO:tasks.rados.rados.0.vpm137.stdout:3530: finishing write tid 3 to vpm1375878-286 2014-08-08T10:49:50.960 INFO:tasks.rados.rados.0.vpm137.stdout:3530: finishing write tid 5 to vpm1375878-286 2014-08-08T10:49:50.961 INFO:tasks.rados.rados.0.vpm137.stdout:3531: finishing write tid 1 to vpm1375878-258 2014-08-08T10:49:50.963 INFO:tasks.rados.rados.0.vpm137.stdout:3528: expect (ObjNum 1751 snap 0 seq_num 1751) 2014-08-08T10:49:50.974 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:50.974 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:51.171 INFO:tasks.rados.rados.0.vpm137.stdout:3527: done (11 left) 2014-08-08T10:49:51.172 INFO:tasks.rados.rados.0.vpm137.stdout:3528: done (10 left) 2014-08-08T10:49:51.172 INFO:tasks.rados.rados.0.vpm137.stdout:3529: done (9 left) 2014-08-08T10:49:51.172 INFO:tasks.rados.rados.0.vpm137.stdout:3537: done (8 left) 2014-08-08T10:49:51.172 INFO:tasks.rados.rados.0.vpm137.stdout:3538: read oid 239 snap -1 2014-08-08T10:49:51.173 INFO:tasks.rados.rados.0.vpm137.stdout:3539: read oid 161 snap -1 2014-08-08T10:49:51.176 INFO:tasks.rados.rados.0.vpm137.stdout:3540: read oid 137 snap -1 2014-08-08T10:49:51.177 INFO:tasks.rados.rados.0.vpm137.stdout:3541: read oid 235 snap -1 2014-08-08T10:49:51.177 INFO:tasks.rados.rados.0.vpm137.stdout:3542: read oid 427 snap -1 2014-08-08T10:49:51.178 INFO:tasks.rados.rados.0.vpm137.stdout:3543: delete oid 386 current snap is 0 2014-08-08T10:49:51.179 INFO:tasks.rados.rados.0.vpm137.stdout:3531: finishing write tid 2 to vpm1375878-258 2014-08-08T10:49:51.179 INFO:tasks.rados.rados.0.vpm137.stdout:3531: finishing write tid 3 to vpm1375878-258 2014-08-08T10:49:51.179 INFO:tasks.rados.rados.0.vpm137.stdout:3531: finishing write tid 5 to vpm1375878-258 2014-08-08T10:49:51.180 INFO:tasks.rados.rados.0.vpm137.stdout:3534: finishing write tid 1 to vpm1375878-408 2014-08-08T10:49:51.180 INFO:tasks.rados.rados.0.vpm137.stdout:3534: finishing write tid 3 to vpm1375878-408 2014-08-08T10:49:51.180 INFO:tasks.rados.rados.0.vpm137.stdout:3536: expect (ObjNum 1338 snap 0 seq_num 1338) 2014-08-08T10:49:51.324 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:52.698 INFO:tasks.rados.rados.0.vpm137.stdout:3526: expect (ObjNum 2031 snap 0 seq_num 2031) 2014-08-08T10:49:53.285 INFO:tasks.rados.rados.0.vpm137.stdout:3530: finishing write tid 6 to vpm1375878-286 2014-08-08T10:49:53.285 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 286 v 334 (ObjNum 2171 snap 0 seq_num 2171) dirty exists 2014-08-08T10:49:53.285 INFO:tasks.rados.rados.0.vpm137.stdout:3532: finishing write tid 1 to vpm1375878-466 2014-08-08T10:49:53.285 INFO:tasks.rados.rados.0.vpm137.stdout:3532: finishing write tid 2 to vpm1375878-466 2014-08-08T10:49:53.285 INFO:tasks.rados.rados.0.vpm137.stdout:3532: finishing write tid 3 to vpm1375878-466 2014-08-08T10:49:53.286 INFO:tasks.rados.rados.0.vpm137.stdout:3532: finishing write tid 5 to vpm1375878-466 2014-08-08T10:49:53.286 INFO:tasks.rados.rados.0.vpm137.stdout:3535: finishing write tid 1 to vpm1375878-129 2014-08-08T10:49:53.286 INFO:tasks.rados.rados.0.vpm137.stdout:3535: finishing write tid 3 to vpm1375878-129 2014-08-08T10:49:53.286 INFO:tasks.rados.rados.0.vpm137.stdout:3531: finishing write tid 6 to vpm1375878-258 2014-08-08T10:49:53.287 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 258 v 374 (ObjNum 2172 snap 0 seq_num 2172) dirty exists 2014-08-08T10:49:53.287 INFO:tasks.rados.rados.0.vpm137.stdout:3534: finishing write tid 4 to vpm1375878-408 2014-08-08T10:49:53.287 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 408 v 345 (ObjNum 2175 snap 0 seq_num 2175) dirty exists 2014-08-08T10:49:53.287 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 1 to vpm1375878-317 2014-08-08T10:49:53.287 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 2 to vpm1375878-317 2014-08-08T10:49:53.288 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 3 to vpm1375878-317 2014-08-08T10:49:53.288 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 4 to vpm1375878-317 2014-08-08T10:49:53.288 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 6 to vpm1375878-317 2014-08-08T10:49:53.288 INFO:tasks.rados.rados.0.vpm137.stdout:3535: finishing write tid 4 to vpm1375878-129 2014-08-08T10:49:53.288 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 129 v 423 (ObjNum 2176 snap 0 seq_num 2176) dirty exists 2014-08-08T10:49:53.425 INFO:tasks.rados.rados.0.vpm137.stdout:3542: expect (ObjNum 770 snap 0 seq_num 770) 2014-08-08T10:49:53.516 INFO:tasks.rados.rados.0.vpm137.stdout:3539: expect (ObjNum 2138 snap 0 seq_num 2138) 2014-08-08T10:49:54.352 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:54.633 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:54.688 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:54.762 INFO:tasks.rados.rados.0.vpm137.stdout:3526: done (13 left) 2014-08-08T10:49:54.762 INFO:tasks.rados.rados.0.vpm137.stdout:3530: done (12 left) 2014-08-08T10:49:54.762 INFO:tasks.rados.rados.0.vpm137.stdout:3531: done (11 left) 2014-08-08T10:49:54.763 INFO:tasks.rados.rados.0.vpm137.stdout:3534: done (10 left) 2014-08-08T10:49:54.763 INFO:tasks.rados.rados.0.vpm137.stdout:3535: done (9 left) 2014-08-08T10:49:54.763 INFO:tasks.rados.rados.0.vpm137.stdout:3536: done (8 left) 2014-08-08T10:49:54.763 INFO:tasks.rados.rados.0.vpm137.stdout:3539: done (7 left) 2014-08-08T10:49:54.764 INFO:tasks.rados.rados.0.vpm137.stdout:3542: done (6 left) 2014-08-08T10:49:54.764 INFO:tasks.rados.rados.0.vpm137.stdout:3543: done (5 left) 2014-08-08T10:49:54.764 INFO:tasks.rados.rados.0.vpm137.stdout:3544: read oid 70 snap -1 2014-08-08T10:49:54.764 INFO:tasks.rados.rados.0.vpm137.stdout:3545: write oid 455 current snap is 0 2014-08-08T10:49:54.764 INFO:tasks.rados.rados.0.vpm137.stdout:3545: seq_num 2179 ranges {679361=503455,1594792=284569} 2014-08-08T10:49:54.785 INFO:tasks.rados.rados.0.vpm137.stdout:3545: writing vpm1375878-455 from 679361 to 1182816 tid 1 2014-08-08T10:49:54.803 INFO:tasks.rados.rados.0.vpm137.stdout:3545: writing vpm1375878-455 from 1594792 to 1879361 tid 2 2014-08-08T10:49:54.805 INFO:tasks.rados.rados.0.vpm137.stdout:3533: finishing write tid 7 to vpm1375878-317 2014-08-08T10:49:54.805 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 317 v 345 (ObjNum 2174 snap 0 seq_num 2174) dirty exists 2014-08-08T10:49:54.805 INFO:tasks.rados.rados.0.vpm137.stdout:3540: expect (ObjNum 1574 snap 0 seq_num 1574) 2014-08-08T10:49:56.149 INFO:tasks.rados.rados.0.vpm137.stdout:3533: done (6 left) 2014-08-08T10:49:56.149 INFO:tasks.rados.rados.0.vpm137.stdout:3540: done (5 left) 2014-08-08T10:49:56.149 INFO:tasks.rados.rados.0.vpm137.stdout:3546: write oid 330 current snap is 0 2014-08-08T10:49:56.149 INFO:tasks.rados.rados.0.vpm137.stdout:3546: seq_num 2180 ranges {623167=475260,1671805=646803,2623166=1} 2014-08-08T10:49:56.179 INFO:tasks.rados.rados.0.vpm137.stdout:3546: writing vpm1375878-330 from 623167 to 1098427 tid 1 2014-08-08T10:49:56.225 INFO:tasks.rados.rados.0.vpm137.stdout:3546: writing vpm1375878-330 from 1671805 to 2318608 tid 2 2014-08-08T10:49:56.232 INFO:tasks.rados.rados.0.vpm137.stdout:3546: writing vpm1375878-330 from 2623166 to 2623167 tid 3 2014-08-08T10:49:56.233 INFO:tasks.rados.rados.0.vpm137.stdout:3532: finishing write tid 6 to vpm1375878-466 2014-08-08T10:49:56.234 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 466 v 399 (ObjNum 2173 snap 0 seq_num 2173) dirty exists 2014-08-08T10:49:56.234 INFO:tasks.rados.rados.0.vpm137.stdout:3541: expect (ObjNum 1970 snap 0 seq_num 1970) 2014-08-08T10:49:56.254 INFO:tasks.rados.rados.0.vpm137.stdout:3532: done (5 left) 2014-08-08T10:49:56.254 INFO:tasks.rados.rados.0.vpm137.stdout:3541: done (4 left) 2014-08-08T10:49:56.254 INFO:tasks.rados.rados.0.vpm137.stdout:3547: write oid 166 current snap is 0 2014-08-08T10:49:56.255 INFO:tasks.rados.rados.0.vpm137.stdout:3547: seq_num 2181 ranges {567996=599512,1786960=468422,2842107=525889} 2014-08-08T10:49:56.298 INFO:tasks.rados.rados.0.vpm137.stdout:3547: writing vpm1375878-166 from 567996 to 1167508 tid 1 2014-08-08T10:49:56.324 INFO:tasks.rados.rados.0.vpm137.stdout:3547: writing vpm1375878-166 from 1786960 to 2255382 tid 2 2014-08-08T10:49:56.349 INFO:tasks.rados.rados.0.vpm137.stdout:3547: writing vpm1375878-166 from 2842107 to 3367996 tid 3 2014-08-08T10:49:56.351 INFO:tasks.rados.rados.0.vpm137.stdout:3538: expect (ObjNum 1976 snap 0 seq_num 1976) 2014-08-08T10:49:57.976 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 824643102, u'bytes_used': 1760976896, u'op_per_sec': 13, u'version': 1669, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1436931, u'write_bytes_sec': 1286608, u'bytes_avail': 427526037504}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:49:57.977 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:49:58.178 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:49:58.179 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:49:58.419 INFO:tasks.rados.rados.0.vpm137.stdout:3538: done (4 left) 2014-08-08T10:49:58.420 INFO:tasks.rados.rados.0.vpm137.stdout:3548: read oid 18 snap -1 2014-08-08T10:49:58.420 INFO:tasks.rados.rados.0.vpm137.stdout:3549: read oid 76 snap -1 2014-08-08T10:49:58.423 INFO:tasks.rados.rados.0.vpm137.stdout:3550: read oid 180 snap -1 2014-08-08T10:49:58.423 INFO:tasks.rados.rados.0.vpm137.stdout:3551: read oid 242 snap -1 2014-08-08T10:49:58.424 INFO:tasks.rados.rados.0.vpm137.stdout:3552: read oid 263 snap -1 2014-08-08T10:49:58.424 INFO:tasks.rados.rados.0.vpm137.stdout:3553: write oid 280 current snap is 0 2014-08-08T10:49:58.425 INFO:tasks.rados.rados.0.vpm137.stdout:3553: seq_num 2182 ranges {780602=571316,2017444=538617,3159685=420917} 2014-08-08T10:49:58.469 INFO:tasks.rados.rados.0.vpm137.stdout:3553: writing vpm1375878-280 from 780602 to 1351918 tid 1 2014-08-08T10:49:58.505 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:49:58.517 INFO:tasks.rados.rados.0.vpm137.stdout:3553: writing vpm1375878-280 from 2017444 to 2556061 tid 2 2014-08-08T10:49:58.536 INFO:tasks.rados.rados.0.vpm137.stdout:3553: writing vpm1375878-280 from 3159685 to 3580602 tid 3 2014-08-08T10:49:58.537 INFO:tasks.rados.rados.0.vpm137.stdout:3545: finishing write tid 1 to vpm1375878-455 2014-08-08T10:49:58.537 INFO:tasks.rados.rados.0.vpm137.stdout:3545: finishing write tid 2 to vpm1375878-455 2014-08-08T10:49:58.538 INFO:tasks.rados.rados.0.vpm137.stdout:3545: finishing write tid 4 to vpm1375878-455 2014-08-08T10:49:58.538 INFO:tasks.rados.rados.0.vpm137.stdout:3545: finishing write tid 5 to vpm1375878-455 2014-08-08T10:49:58.538 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 455 v 424 (ObjNum 2179 snap 0 seq_num 2179) dirty exists 2014-08-08T10:49:58.538 INFO:tasks.rados.rados.0.vpm137.stdout:3546: finishing write tid 1 to vpm1375878-330 2014-08-08T10:49:58.538 INFO:tasks.rados.rados.0.vpm137.stdout:3546: finishing write tid 2 to vpm1375878-330 2014-08-08T10:49:58.539 INFO:tasks.rados.rados.0.vpm137.stdout:3546: finishing write tid 3 to vpm1375878-330 2014-08-08T10:49:58.539 INFO:tasks.rados.rados.0.vpm137.stdout:3546: finishing write tid 5 to vpm1375878-330 2014-08-08T10:49:58.539 INFO:tasks.rados.rados.0.vpm137.stdout:3547: finishing write tid 1 to vpm1375878-166 2014-08-08T10:49:58.539 INFO:tasks.rados.rados.0.vpm137.stdout:3547: finishing write tid 2 to vpm1375878-166 2014-08-08T10:49:58.539 INFO:tasks.rados.rados.0.vpm137.stdout:3547: finishing write tid 3 to vpm1375878-166 2014-08-08T10:49:58.540 INFO:tasks.rados.rados.0.vpm137.stdout:3547: finishing write tid 5 to vpm1375878-166 2014-08-08T10:49:58.541 INFO:tasks.rados.rados.0.vpm137.stdout:3546: finishing write tid 6 to vpm1375878-330 2014-08-08T10:49:58.541 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 330 v 427 (ObjNum 2180 snap 0 seq_num 2180) dirty exists 2014-08-08T10:49:58.541 INFO:tasks.rados.rados.0.vpm137.stdout:3547: finishing write tid 6 to vpm1375878-166 2014-08-08T10:49:58.541 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 166 v 431 (ObjNum 2181 snap 0 seq_num 2181) dirty exists 2014-08-08T10:49:58.541 INFO:tasks.rados.rados.0.vpm137.stdout:3550: expect (ObjNum 1442 snap 0 seq_num 1442) 2014-08-08T10:49:58.877 INFO:tasks.rados.rados.0.vpm137.stdout:3544: done (9 left) 2014-08-08T10:49:58.878 INFO:tasks.rados.rados.0.vpm137.stdout:3545: done (8 left) 2014-08-08T10:49:58.878 INFO:tasks.rados.rados.0.vpm137.stdout:3546: done (7 left) 2014-08-08T10:49:58.878 INFO:tasks.rados.rados.0.vpm137.stdout:3547: done (6 left) 2014-08-08T10:49:58.878 INFO:tasks.rados.rados.0.vpm137.stdout:3549: done (5 left) 2014-08-08T10:49:58.878 INFO:tasks.rados.rados.0.vpm137.stdout:3550: done (4 left) 2014-08-08T10:49:58.879 INFO:tasks.rados.rados.0.vpm137.stdout:3554: write oid 34 current snap is 0 2014-08-08T10:49:58.879 INFO:tasks.rados.rados.0.vpm137.stdout:3554: seq_num 2183 ranges {475832=578193,1730554=608812,3093136=182696} 2014-08-08T10:49:58.912 INFO:tasks.rados.rados.0.vpm137.stdout:3554: writing vpm1375878-34 from 475832 to 1054025 tid 1 2014-08-08T10:49:58.955 INFO:tasks.rados.rados.0.vpm137.stdout:3554: writing vpm1375878-34 from 1730554 to 2339366 tid 2 2014-08-08T10:49:58.976 INFO:tasks.rados.rados.0.vpm137.stdout:3554: writing vpm1375878-34 from 3093136 to 3275832 tid 3 2014-08-08T10:49:58.978 INFO:tasks.rados.rados.0.vpm137.stdout:3548: expect (ObjNum 2070 snap 0 seq_num 2070) 2014-08-08T10:49:59.503 INFO:tasks.rados.rados.0.vpm137.stdout:3548: done (4 left) 2014-08-08T10:49:59.503 INFO:tasks.rados.rados.0.vpm137.stdout:3555: write oid 88 current snap is 0 2014-08-08T10:49:59.503 INFO:tasks.rados.rados.0.vpm137.stdout:3555: seq_num 2184 ranges {669238=400000} 2014-08-08T10:49:59.522 INFO:tasks.rados.rados.0.vpm137.stdout:3555: writing vpm1375878-88 from 669238 to 1069238 tid 1 2014-08-08T10:49:59.523 INFO:tasks.rados.rados.0.vpm137.stdout:3551: expect (ObjNum 1790 snap 0 seq_num 1790) 2014-08-08T10:50:01.540 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:01.804 INFO:tasks.rados.rados.0.vpm137.stdout:3551: done (4 left) 2014-08-08T10:50:01.805 INFO:tasks.rados.rados.0.vpm137.stdout:3552: done (3 left) 2014-08-08T10:50:01.805 INFO:tasks.rados.rados.0.vpm137.stdout:3556: delete oid 62 current snap is 0 2014-08-08T10:50:01.806 INFO:tasks.rados.rados.0.vpm137.stdout:3553: finishing write tid 1 to vpm1375878-280 2014-08-08T10:50:01.807 INFO:tasks.rados.rados.0.vpm137.stdout:3553: finishing write tid 2 to vpm1375878-280 2014-08-08T10:50:01.807 INFO:tasks.rados.rados.0.vpm137.stdout:3553: finishing write tid 3 to vpm1375878-280 2014-08-08T10:50:01.807 INFO:tasks.rados.rados.0.vpm137.stdout:3553: finishing write tid 5 to vpm1375878-280 2014-08-08T10:50:01.807 INFO:tasks.rados.rados.0.vpm137.stdout:3553: finishing write tid 6 to vpm1375878-280 2014-08-08T10:50:01.809 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 280 v 427 (ObjNum 2182 snap 0 seq_num 2182) dirty exists 2014-08-08T10:50:01.809 INFO:tasks.rados.rados.0.vpm137.stdout:3554: finishing write tid 1 to vpm1375878-34 2014-08-08T10:50:01.809 INFO:tasks.rados.rados.0.vpm137.stdout:3554: finishing write tid 2 to vpm1375878-34 2014-08-08T10:50:01.810 INFO:tasks.rados.rados.0.vpm137.stdout:3554: finishing write tid 3 to vpm1375878-34 2014-08-08T10:50:01.810 INFO:tasks.rados.rados.0.vpm137.stdout:3554: finishing write tid 5 to vpm1375878-34 2014-08-08T10:50:01.810 INFO:tasks.rados.rados.0.vpm137.stdout:3554: finishing write tid 6 to vpm1375878-34 2014-08-08T10:50:01.810 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 34 v 401 (ObjNum 2183 snap 0 seq_num 2183) dirty exists 2014-08-08T10:50:01.811 INFO:tasks.rados.rados.0.vpm137.stdout:3555: finishing write tid 1 to vpm1375878-88 2014-08-08T10:50:01.811 INFO:tasks.rados.rados.0.vpm137.stdout:3555: finishing write tid 3 to vpm1375878-88 2014-08-08T10:50:01.811 INFO:tasks.rados.rados.0.vpm137.stdout:3555: finishing write tid 4 to vpm1375878-88 2014-08-08T10:50:01.811 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 88 v 411 (ObjNum 2184 snap 0 seq_num 2184) dirty exists 2014-08-08T10:50:01.820 INFO:tasks.rados.rados.0.vpm137.stdout:3553: done (3 left) 2014-08-08T10:50:01.820 INFO:tasks.rados.rados.0.vpm137.stdout:3554: done (2 left) 2014-08-08T10:50:01.820 INFO:tasks.rados.rados.0.vpm137.stdout:3555: done (1 left) 2014-08-08T10:50:01.821 INFO:tasks.rados.rados.0.vpm137.stdout:3556: done (0 left) 2014-08-08T10:50:01.821 INFO:tasks.rados.rados.0.vpm137.stdout:3557: read oid 374 snap -1 2014-08-08T10:50:01.821 INFO:tasks.rados.rados.0.vpm137.stdout:3558: read oid 85 snap -1 2014-08-08T10:50:01.821 INFO:tasks.rados.rados.0.vpm137.stdout:3559: write oid 135 current snap is 0 2014-08-08T10:50:01.822 INFO:tasks.rados.rados.0.vpm137.stdout:3559: seq_num 2186 ranges {459696=1} 2014-08-08T10:50:01.826 INFO:tasks.rados.rados.0.vpm137.stdout:3559: writing vpm1375878-135 from 459696 to 459697 tid 1 2014-08-08T10:50:01.827 INFO:tasks.rados.rados.0.vpm137.stdout:3560: write oid 274 current snap is 0 2014-08-08T10:50:01.828 INFO:tasks.rados.rados.0.vpm137.stdout:3560: seq_num 2187 ranges {769455=652930,1847114=641017,2769454=1} 2014-08-08T10:50:01.870 INFO:tasks.rados.rados.0.vpm137.stdout:3560: writing vpm1375878-274 from 769455 to 1422385 tid 1 2014-08-08T10:50:01.913 INFO:tasks.rados.rados.0.vpm137.stdout:3560: writing vpm1375878-274 from 1847114 to 2488131 tid 2 2014-08-08T10:50:01.917 INFO:tasks.rados.rados.0.vpm137.stdout:3560: writing vpm1375878-274 from 2769454 to 2769455 tid 3 2014-08-08T10:50:01.918 INFO:tasks.rados.rados.0.vpm137.stdout:3559: finishing write tid 1 to vpm1375878-135 2014-08-08T10:50:01.918 INFO:tasks.rados.rados.0.vpm137.stdout:3559: finishing write tid 3 to vpm1375878-135 2014-08-08T10:50:01.918 INFO:tasks.rados.rados.0.vpm137.stdout:3559: finishing write tid 4 to vpm1375878-135 2014-08-08T10:50:01.919 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 135 v 342 (ObjNum 2186 snap 0 seq_num 2186) dirty exists 2014-08-08T10:50:01.919 INFO:tasks.rados.rados.0.vpm137.stdout:3559: done (3 left) 2014-08-08T10:50:01.919 INFO:tasks.rados.rados.0.vpm137.stdout:3561: read oid 96 snap -1 2014-08-08T10:50:01.920 INFO:tasks.rados.rados.0.vpm137.stdout:3562: write oid 291 current snap is 0 2014-08-08T10:50:01.920 INFO:tasks.rados.rados.0.vpm137.stdout:3562: seq_num 2188 ranges {464685=625759,1677599=746284,2464684=1} 2014-08-08T10:50:01.999 INFO:tasks.rados.rados.0.vpm137.stdout:3562: writing vpm1375878-291 from 464685 to 1090444 tid 1 2014-08-08T10:50:02.049 INFO:tasks.rados.rados.0.vpm137.stdout:3562: writing vpm1375878-291 from 1677599 to 2423883 tid 2 2014-08-08T10:50:02.050 INFO:tasks.rados.rados.0.vpm137.stdout:3562: writing vpm1375878-291 from 2464684 to 2464685 tid 3 2014-08-08T10:50:02.051 INFO:tasks.rados.rados.0.vpm137.stdout:3557: expect (ObjNum 1584 snap 0 seq_num 1584) 2014-08-08T10:50:03.591 INFO:tasks.rados.rados.0.vpm137.stdout:3557: done (4 left) 2014-08-08T10:50:03.591 INFO:tasks.rados.rados.0.vpm137.stdout:3563: read oid 338 snap -1 2014-08-08T10:50:03.592 INFO:tasks.rados.rados.0.vpm137.stdout:3564: read oid 320 snap -1 2014-08-08T10:50:03.593 INFO:tasks.rados.rados.0.vpm137.stdout:3565: write oid 485 current snap is 0 2014-08-08T10:50:03.593 INFO:tasks.rados.rados.0.vpm137.stdout:3565: seq_num 2189 ranges {560938=597563,1640306=520632} 2014-08-08T10:50:03.628 INFO:tasks.rados.rados.0.vpm137.stdout:3565: writing vpm1375878-485 from 560938 to 1158501 tid 1 2014-08-08T10:50:03.654 INFO:tasks.rados.rados.0.vpm137.stdout:3565: writing vpm1375878-485 from 1640306 to 2160938 tid 2 2014-08-08T10:50:03.655 INFO:tasks.rados.rados.0.vpm137.stdout:3560: finishing write tid 1 to vpm1375878-274 2014-08-08T10:50:03.656 INFO:tasks.rados.rados.0.vpm137.stdout:3558: expect (ObjNum 2090 snap 0 seq_num 2090) 2014-08-08T10:50:04.737 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:04.784 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:04.918 INFO:tasks.rados.rados.0.vpm137.stdout:3558: done (6 left) 2014-08-08T10:50:04.918 INFO:tasks.rados.rados.0.vpm137.stdout:3566: read oid 336 snap -1 2014-08-08T10:50:04.919 INFO:tasks.rados.rados.0.vpm137.stdout:3567: read oid 219 snap -1 2014-08-08T10:50:04.919 INFO:tasks.rados.rados.0.vpm137.stdout:3568: write oid 469 current snap is 0 2014-08-08T10:50:04.920 INFO:tasks.rados.rados.0.vpm137.stdout:3568: seq_num 2190 ranges {754344=569367,1967942=486675,2944593=696525,3954343=1} 2014-08-08T10:50:04.966 INFO:tasks.rados.rados.0.vpm137.stdout:3568: writing vpm1375878-469 from 754344 to 1323711 tid 1 2014-08-08T10:50:04.991 INFO:tasks.rados.rados.0.vpm137.stdout:3568: writing vpm1375878-469 from 1967942 to 2454617 tid 2 2014-08-08T10:50:05.018 INFO:tasks.rados.rados.0.vpm137.stdout:3568: writing vpm1375878-469 from 2944593 to 3641118 tid 3 2014-08-08T10:50:05.023 INFO:tasks.rados.rados.0.vpm137.stdout:3568: writing vpm1375878-469 from 3954343 to 3954344 tid 4 2014-08-08T10:50:05.024 INFO:tasks.rados.rados.0.vpm137.stdout:3560: finishing write tid 2 to vpm1375878-274 2014-08-08T10:50:05.024 INFO:tasks.rados.rados.0.vpm137.stdout:3560: finishing write tid 3 to vpm1375878-274 2014-08-08T10:50:05.025 INFO:tasks.rados.rados.0.vpm137.stdout:3561: expect (ObjNum 1567 snap 0 seq_num 1567) 2014-08-08T10:50:05.256 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 826688916, u'bytes_used': 1764052992, u'op_per_sec': 14, u'version': 1670, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2124765, u'write_bytes_sec': 1304728, u'bytes_avail': 427522961408}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:05.256 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:05.457 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:05.458 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:05.676 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:06.403 INFO:tasks.rados.rados.0.vpm137.stdout:3561: done (8 left) 2014-08-08T10:50:06.403 INFO:tasks.rados.rados.0.vpm137.stdout:3569: read oid 484 snap -1 2014-08-08T10:50:06.403 INFO:tasks.rados.rados.0.vpm137.stdout:3570: write oid 60 current snap is 0 2014-08-08T10:50:06.403 INFO:tasks.rados.rados.0.vpm137.stdout:3570: seq_num 2191 ranges {165925=1} 2014-08-08T10:50:06.405 INFO:tasks.rados.rados.0.vpm137.stdout:3570: writing vpm1375878-60 from 165925 to 165926 tid 1 2014-08-08T10:50:06.406 INFO:tasks.rados.rados.0.vpm137.stdout:3571: write oid 393 current snap is 0 2014-08-08T10:50:06.406 INFO:tasks.rados.rados.0.vpm137.stdout:3571: seq_num 2192 ranges {662179=549072,1796206=529913,2984163=717658,3862178=1} 2014-08-08T10:50:06.427 INFO:tasks.rados.rados.0.vpm137.stdout:3571: writing vpm1375878-393 from 662179 to 1211251 tid 1 2014-08-08T10:50:06.453 INFO:tasks.rados.rados.0.vpm137.stdout:3571: writing vpm1375878-393 from 1796206 to 2326119 tid 2 2014-08-08T10:50:06.481 INFO:tasks.rados.rados.0.vpm137.stdout:3571: writing vpm1375878-393 from 2984163 to 3701821 tid 3 2014-08-08T10:50:06.483 INFO:tasks.rados.rados.0.vpm137.stdout:3571: writing vpm1375878-393 from 3862178 to 3862179 tid 4 2014-08-08T10:50:06.484 INFO:tasks.rados.rados.0.vpm137.stdout:3560: finishing write tid 5 to vpm1375878-274 2014-08-08T10:50:06.484 INFO:tasks.rados.rados.0.vpm137.stdout:3560: finishing write tid 6 to vpm1375878-274 2014-08-08T10:50:06.485 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 274 v 349 (ObjNum 2187 snap 0 seq_num 2187) dirty exists 2014-08-08T10:50:06.485 INFO:tasks.rados.rados.0.vpm137.stdout:3562: finishing write tid 1 to vpm1375878-291 2014-08-08T10:50:06.485 INFO:tasks.rados.rados.0.vpm137.stdout:3562: finishing write tid 2 to vpm1375878-291 2014-08-08T10:50:06.485 INFO:tasks.rados.rados.0.vpm137.stdout:3562: finishing write tid 3 to vpm1375878-291 2014-08-08T10:50:06.485 INFO:tasks.rados.rados.0.vpm137.stdout:3562: finishing write tid 5 to vpm1375878-291 2014-08-08T10:50:06.486 INFO:tasks.rados.rados.0.vpm137.stdout:3562: finishing write tid 6 to vpm1375878-291 2014-08-08T10:50:06.486 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 291 v 339 (ObjNum 2188 snap 0 seq_num 2188) dirty exists 2014-08-08T10:50:06.486 INFO:tasks.rados.rados.0.vpm137.stdout:3564: expect (ObjNum 1946 snap 0 seq_num 1946) 2014-08-08T10:50:06.649 INFO:tasks.rados.rados.0.vpm137.stdout:3560: done (10 left) 2014-08-08T10:50:06.649 INFO:tasks.rados.rados.0.vpm137.stdout:3562: done (9 left) 2014-08-08T10:50:06.649 INFO:tasks.rados.rados.0.vpm137.stdout:3564: done (8 left) 2014-08-08T10:50:06.650 INFO:tasks.rados.rados.0.vpm137.stdout:3572: read oid 437 snap -1 2014-08-08T10:50:06.650 INFO:tasks.rados.rados.0.vpm137.stdout:3573: write oid 359 current snap is 0 2014-08-08T10:50:06.650 INFO:tasks.rados.rados.0.vpm137.stdout:3573: seq_num 2193 ranges {453537=672300,1653536=1} 2014-08-08T10:50:06.678 INFO:tasks.rados.rados.0.vpm137.stdout:3573: writing vpm1375878-359 from 453537 to 1125837 tid 1 2014-08-08T10:50:06.685 INFO:tasks.rados.rados.0.vpm137.stdout:3573: writing vpm1375878-359 from 1653536 to 1653537 tid 2 2014-08-08T10:50:06.686 INFO:tasks.rados.rados.0.vpm137.stdout:3563: expect (ObjNum 1698 snap 0 seq_num 1698) 2014-08-08T10:50:08.014 INFO:tasks.rados.rados.0.vpm137.stdout:3563: done (9 left) 2014-08-08T10:50:08.015 INFO:tasks.rados.rados.0.vpm137.stdout:3574: read oid 319 snap -1 2014-08-08T10:50:08.016 INFO:tasks.rados.rados.0.vpm137.stdout:3575: delete oid 475 current snap is 0 2014-08-08T10:50:08.016 INFO:tasks.rados.rados.0.vpm137.stdout:3565: finishing write tid 1 to vpm1375878-485 2014-08-08T10:50:08.016 INFO:tasks.rados.rados.0.vpm137.stdout:3565: finishing write tid 2 to vpm1375878-485 2014-08-08T10:50:08.017 INFO:tasks.rados.rados.0.vpm137.stdout:3565: finishing write tid 4 to vpm1375878-485 2014-08-08T10:50:08.017 INFO:tasks.rados.rados.0.vpm137.stdout:3565: finishing write tid 5 to vpm1375878-485 2014-08-08T10:50:08.018 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 485 v 591 (ObjNum 2189 snap 0 seq_num 2189) dirty exists 2014-08-08T10:50:08.018 INFO:tasks.rados.rados.0.vpm137.stdout:3566: expect (ObjNum 2101 snap 0 seq_num 2101) 2014-08-08T10:50:08.706 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:09.999 INFO:tasks.rados.rados.0.vpm137.stdout:3565: done (10 left) 2014-08-08T10:50:10.000 INFO:tasks.rados.rados.0.vpm137.stdout:3566: done (9 left) 2014-08-08T10:50:10.000 INFO:tasks.rados.rados.0.vpm137.stdout:3575: done (8 left) 2014-08-08T10:50:10.000 INFO:tasks.rados.rados.0.vpm137.stdout:3576: read oid 475 snap -1 2014-08-08T10:50:10.001 INFO:tasks.rados.rados.0.vpm137.stdout:3577: write oid 140 current snap is 0 2014-08-08T10:50:10.001 INFO:tasks.rados.rados.0.vpm137.stdout:3577: seq_num 2195 ranges {646045=400000} 2014-08-08T10:50:10.030 INFO:tasks.rados.rados.0.vpm137.stdout:3577: writing vpm1375878-140 from 646045 to 1046045 tid 1 2014-08-08T10:50:10.034 INFO:tasks.rados.rados.0.vpm137.stdout:3567: expect (ObjNum 1862 snap 0 seq_num 1862) 2014-08-08T10:50:11.910 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:11.962 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:12.203 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:49:13.453146', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89752584, u'health': u'HEALTH_OK', u'kb_used': 9200984, u'store_stats': {u'bytes_total': 24546482, u'bytes_log': 5124622, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:49:17.561610', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89751940, u'health': u'HEALTH_OK', u'kb_used': 9201628, u'store_stats': {u'bytes_total': 27101962, u'bytes_log': 7680158, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 829696363, u'bytes_used': 1764732928, u'op_per_sec': 10, u'version': 1672, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2355743, u'write_bytes_sec': 841717, u'bytes_avail': 427522281472}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:12.204 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:12.420 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:12.420 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:12.429 INFO:tasks.rados.rados.0.vpm137.stdout:3567: done (9 left) 2014-08-08T10:50:12.430 INFO:tasks.rados.rados.0.vpm137.stdout:3578: read oid 133 snap -1 2014-08-08T10:50:12.431 INFO:tasks.rados.rados.0.vpm137.stdout:3579: write oid 324 current snap is 0 2014-08-08T10:50:12.431 INFO:tasks.rados.rados.0.vpm137.stdout:3579: seq_num 2196 ranges {706202=623809,1779518=526684} 2014-08-08T10:50:12.476 INFO:tasks.rados.rados.0.vpm137.stdout:3579: writing vpm1375878-324 from 706202 to 1330011 tid 1 2014-08-08T10:50:12.519 INFO:tasks.rados.rados.0.vpm137.stdout:3579: writing vpm1375878-324 from 1779518 to 2306202 tid 2 2014-08-08T10:50:12.522 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 1 to vpm1375878-469 2014-08-08T10:50:12.522 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 2 to vpm1375878-469 2014-08-08T10:50:12.522 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 3 to vpm1375878-469 2014-08-08T10:50:12.523 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 4 to vpm1375878-469 2014-08-08T10:50:12.523 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 6 to vpm1375878-469 2014-08-08T10:50:12.523 INFO:tasks.rados.rados.0.vpm137.stdout:3568: finishing write tid 7 to vpm1375878-469 2014-08-08T10:50:12.524 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 469 v 379 (ObjNum 2190 snap 0 seq_num 2190) dirty exists 2014-08-08T10:50:12.524 INFO:tasks.rados.rados.0.vpm137.stdout:3570: finishing write tid 1 to vpm1375878-60 2014-08-08T10:50:12.524 INFO:tasks.rados.rados.0.vpm137.stdout:3570: finishing write tid 3 to vpm1375878-60 2014-08-08T10:50:12.528 INFO:tasks.rados.rados.0.vpm137.stdout:3570: finishing write tid 4 to vpm1375878-60 2014-08-08T10:50:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 60 v 426 (ObjNum 2191 snap 0 seq_num 2191) dirty exists 2014-08-08T10:50:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 1 to vpm1375878-393 2014-08-08T10:50:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 2 to vpm1375878-393 2014-08-08T10:50:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 3 to vpm1375878-393 2014-08-08T10:50:12.530 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 4 to vpm1375878-393 2014-08-08T10:50:12.530 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 6 to vpm1375878-393 2014-08-08T10:50:12.530 INFO:tasks.rados.rados.0.vpm137.stdout:3571: finishing write tid 7 to vpm1375878-393 2014-08-08T10:50:12.530 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 393 v 301 (ObjNum 2192 snap 0 seq_num 2192) dirty exists 2014-08-08T10:50:12.531 INFO:tasks.rados.rados.0.vpm137.stdout:3573: finishing write tid 1 to vpm1375878-359 2014-08-08T10:50:12.531 INFO:tasks.rados.rados.0.vpm137.stdout:3573: finishing write tid 2 to vpm1375878-359 2014-08-08T10:50:12.531 INFO:tasks.rados.rados.0.vpm137.stdout:3573: finishing write tid 4 to vpm1375878-359 2014-08-08T10:50:12.531 INFO:tasks.rados.rados.0.vpm137.stdout:3572: expect (ObjNum 1904 snap 0 seq_num 1904) 2014-08-08T10:50:12.734 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:14.116 INFO:tasks.rados.rados.0.vpm137.stdout:3568: done (10 left) 2014-08-08T10:50:14.117 INFO:tasks.rados.rados.0.vpm137.stdout:3569: done (9 left) 2014-08-08T10:50:14.117 INFO:tasks.rados.rados.0.vpm137.stdout:3570: done (8 left) 2014-08-08T10:50:14.117 INFO:tasks.rados.rados.0.vpm137.stdout:3571: done (7 left) 2014-08-08T10:50:14.118 INFO:tasks.rados.rados.0.vpm137.stdout:3572: done (6 left) 2014-08-08T10:50:14.118 INFO:tasks.rados.rados.0.vpm137.stdout:3580: write oid 468 current snap is 0 2014-08-08T10:50:14.118 INFO:tasks.rados.rados.0.vpm137.stdout:3580: seq_num 2197 ranges {651032=747038,1893651=514937,3035794=15238} 2014-08-08T10:50:14.148 INFO:tasks.rados.rados.0.vpm137.stdout:3580: writing vpm1375878-468 from 651032 to 1398070 tid 1 2014-08-08T10:50:14.174 INFO:tasks.rados.rados.0.vpm137.stdout:3580: writing vpm1375878-468 from 1893651 to 2408588 tid 2 2014-08-08T10:50:14.183 INFO:tasks.rados.rados.0.vpm137.stdout:3580: writing vpm1375878-468 from 3035794 to 3051032 tid 3 2014-08-08T10:50:14.184 INFO:tasks.rados.rados.0.vpm137.stdout:3573: finishing write tid 5 to vpm1375878-359 2014-08-08T10:50:14.184 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 359 v 352 (ObjNum 2193 snap 0 seq_num 2193) dirty exists 2014-08-08T10:50:14.185 INFO:tasks.rados.rados.0.vpm137.stdout:3574: expect (ObjNum 1449 snap 0 seq_num 1449) 2014-08-08T10:50:14.601 INFO:tasks.rados.rados.0.vpm137.stdout:3573: done (6 left) 2014-08-08T10:50:14.601 INFO:tasks.rados.rados.0.vpm137.stdout:3574: done (5 left) 2014-08-08T10:50:14.602 INFO:tasks.rados.rados.0.vpm137.stdout:3581: read oid 481 snap -1 2014-08-08T10:50:14.602 INFO:tasks.rados.rados.0.vpm137.stdout:3582: write oid 95 current snap is 0 2014-08-08T10:50:14.603 INFO:tasks.rados.rados.0.vpm137.stdout:3582: seq_num 2198 ranges {594838=719866,1821287=585132,3050523=510209,3794837=1} 2014-08-08T10:50:14.638 INFO:tasks.rados.rados.0.vpm137.stdout:3582: writing vpm1375878-95 from 594838 to 1314704 tid 1 2014-08-08T10:50:14.667 INFO:tasks.rados.rados.0.vpm137.stdout:3582: writing vpm1375878-95 from 1821287 to 2406419 tid 2 2014-08-08T10:50:14.690 INFO:tasks.rados.rados.0.vpm137.stdout:3582: writing vpm1375878-95 from 3050523 to 3560732 tid 3 2014-08-08T10:50:14.693 INFO:tasks.rados.rados.0.vpm137.stdout:3582: writing vpm1375878-95 from 3794837 to 3794838 tid 4 2014-08-08T10:50:14.694 INFO:tasks.rados.rados.0.vpm137.stdout:3577: finishing write tid 1 to vpm1375878-140 2014-08-08T10:50:14.694 INFO:tasks.rados.rados.0.vpm137.stdout:3577: finishing write tid 3 to vpm1375878-140 2014-08-08T10:50:14.695 INFO:tasks.rados.rados.0.vpm137.stdout:3577: finishing write tid 4 to vpm1375878-140 2014-08-08T10:50:14.695 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 140 v 344 (ObjNum 2195 snap 0 seq_num 2195) dirty exists 2014-08-08T10:50:14.695 INFO:tasks.rados.rados.0.vpm137.stdout:3578: expect (ObjNum 870 snap 0 seq_num 870) 2014-08-08T10:50:14.852 INFO:tasks.rados.rados.0.vpm137.stdout:3576: done (6 left) 2014-08-08T10:50:14.853 INFO:tasks.rados.rados.0.vpm137.stdout:3577: done (5 left) 2014-08-08T10:50:14.853 INFO:tasks.rados.rados.0.vpm137.stdout:3578: done (4 left) 2014-08-08T10:50:14.853 INFO:tasks.rados.rados.0.vpm137.stdout:3583: delete oid 157 current snap is 0 2014-08-08T10:50:14.853 INFO:tasks.rados.rados.0.vpm137.stdout:3579: finishing write tid 1 to vpm1375878-324 2014-08-08T10:50:14.854 INFO:tasks.rados.rados.0.vpm137.stdout:3579: finishing write tid 2 to vpm1375878-324 2014-08-08T10:50:14.854 INFO:tasks.rados.rados.0.vpm137.stdout:3579: finishing write tid 4 to vpm1375878-324 2014-08-08T10:50:14.855 INFO:tasks.rados.rados.0.vpm137.stdout:3579: finishing write tid 5 to vpm1375878-324 2014-08-08T10:50:14.855 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 324 v 414 (ObjNum 2196 snap 0 seq_num 2196) dirty exists 2014-08-08T10:50:14.855 INFO:tasks.rados.rados.0.vpm137.stdout:3580: finishing write tid 1 to vpm1375878-468 2014-08-08T10:50:14.855 INFO:tasks.rados.rados.0.vpm137.stdout:3580: finishing write tid 2 to vpm1375878-468 2014-08-08T10:50:14.856 INFO:tasks.rados.rados.0.vpm137.stdout:3580: finishing write tid 3 to vpm1375878-468 2014-08-08T10:50:14.856 INFO:tasks.rados.rados.0.vpm137.stdout:3580: finishing write tid 5 to vpm1375878-468 2014-08-08T10:50:14.857 INFO:tasks.rados.rados.0.vpm137.stdout:3580: finishing write tid 6 to vpm1375878-468 2014-08-08T10:50:14.857 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 468 v 389 (ObjNum 2197 snap 0 seq_num 2197) dirty exists 2014-08-08T10:50:14.857 INFO:tasks.rados.rados.0.vpm137.stdout:3581: expect (ObjNum 2154 snap 0 seq_num 2154) 2014-08-08T10:50:15.696 INFO:tasks.rados.rados.0.vpm137.stdout:3579: done (4 left) 2014-08-08T10:50:15.696 INFO:tasks.rados.rados.0.vpm137.stdout:3580: done (3 left) 2014-08-08T10:50:15.696 INFO:tasks.rados.rados.0.vpm137.stdout:3581: done (2 left) 2014-08-08T10:50:15.697 INFO:tasks.rados.rados.0.vpm137.stdout:3583: done (1 left) 2014-08-08T10:50:15.697 INFO:tasks.rados.rados.0.vpm137.stdout:3584: write oid 389 current snap is 0 2014-08-08T10:50:15.697 INFO:tasks.rados.rados.0.vpm137.stdout:3584: seq_num 2200 ranges {752272=1} 2014-08-08T10:50:15.705 INFO:tasks.rados.rados.0.vpm137.stdout:3584: writing vpm1375878-389 from 752272 to 752273 tid 1 2014-08-08T10:50:15.707 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 1 to vpm1375878-95 2014-08-08T10:50:15.707 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 2 to vpm1375878-95 2014-08-08T10:50:15.707 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 3 to vpm1375878-95 2014-08-08T10:50:15.707 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 4 to vpm1375878-95 2014-08-08T10:50:15.707 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 6 to vpm1375878-95 2014-08-08T10:50:15.708 INFO:tasks.rados.rados.0.vpm137.stdout:3582: finishing write tid 7 to vpm1375878-95 2014-08-08T10:50:15.708 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 95 v 596 (ObjNum 2198 snap 0 seq_num 2198) dirty exists 2014-08-08T10:50:15.709 INFO:tasks.rados.rados.0.vpm137.stdout:3582: done (1 left) 2014-08-08T10:50:15.709 INFO:tasks.rados.rados.0.vpm137.stdout:3585: delete oid 87 current snap is 0 2014-08-08T10:50:15.727 INFO:tasks.rados.rados.0.vpm137.stdout:3584: finishing write tid 1 to vpm1375878-389 2014-08-08T10:50:15.730 INFO:tasks.rados.rados.0.vpm137.stdout:3584: finishing write tid 3 to vpm1375878-389 2014-08-08T10:50:15.731 INFO:tasks.rados.rados.0.vpm137.stdout:3584: finishing write tid 4 to vpm1375878-389 2014-08-08T10:50:15.731 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 389 v 416 (ObjNum 2200 snap 0 seq_num 2200) dirty exists 2014-08-08T10:50:15.742 INFO:tasks.rados.rados.0.vpm137.stdout:3584: done (1 left) 2014-08-08T10:50:15.742 INFO:tasks.rados.rados.0.vpm137.stdout:3585: done (0 left) 2014-08-08T10:50:15.743 INFO:tasks.rados.rados.0.vpm137.stdout:3586: read oid 307 snap -1 2014-08-08T10:50:15.743 INFO:tasks.rados.rados.0.vpm137.stdout:3587: read oid 482 snap -1 2014-08-08T10:50:15.744 INFO:tasks.rados.rados.0.vpm137.stdout:3588: delete oid 207 current snap is 0 2014-08-08T10:50:15.759 INFO:tasks.rados.rados.0.vpm137.stdout:3586: expect (ObjNum 616 snap 0 seq_num 616) 2014-08-08T10:50:15.770 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:15.985 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:16.009 INFO:tasks.rados.rados.0.vpm137.stdout:3586: done (2 left) 2014-08-08T10:50:16.009 INFO:tasks.rados.rados.0.vpm137.stdout:3588: done (1 left) 2014-08-08T10:50:16.009 INFO:tasks.rados.rados.0.vpm137.stdout:3589: write oid 52 current snap is 0 2014-08-08T10:50:16.010 INFO:tasks.rados.rados.0.vpm137.stdout:3589: seq_num 2203 ranges {735115=767432,1935114=1} 2014-08-08T10:50:16.031 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:16.042 INFO:tasks.rados.rados.0.vpm137.stdout:3589: writing vpm1375878-52 from 735115 to 1502547 tid 1 2014-08-08T10:50:16.053 INFO:tasks.rados.rados.0.vpm137.stdout:3589: writing vpm1375878-52 from 1935114 to 1935115 tid 2 2014-08-08T10:50:16.054 INFO:tasks.rados.rados.0.vpm137.stdout:3587: done (1 left) 2014-08-08T10:50:16.054 INFO:tasks.rados.rados.0.vpm137.stdout:3590: read oid 368 snap -1 2014-08-08T10:50:16.055 INFO:tasks.rados.rados.0.vpm137.stdout:3591: write oid 360 current snap is 0 2014-08-08T10:50:16.055 INFO:tasks.rados.rados.0.vpm137.stdout:3591: seq_num 2204 ranges {679944=622884,1863264=509151,2679943=1} 2014-08-08T10:50:16.079 INFO:tasks.rados.rados.0.vpm137.stdout:3591: writing vpm1375878-360 from 679944 to 1302828 tid 1 2014-08-08T10:50:16.102 INFO:tasks.rados.rados.0.vpm137.stdout:3591: writing vpm1375878-360 from 1863264 to 2372415 tid 2 2014-08-08T10:50:16.106 INFO:tasks.rados.rados.0.vpm137.stdout:3591: writing vpm1375878-360 from 2679943 to 2679944 tid 3 2014-08-08T10:50:16.111 INFO:tasks.rados.rados.0.vpm137.stdout:3592: read oid 42 snap -1 2014-08-08T10:50:16.113 INFO:tasks.rados.rados.0.vpm137.stdout:3593: read oid 184 snap -1 2014-08-08T10:50:16.114 INFO:tasks.rados.rados.0.vpm137.stdout:3594: delete oid 228 current snap is 0 2014-08-08T10:50:16.215 INFO:tasks.rados.rados.0.vpm137.stdout:3593: expect (ObjNum 1772 snap 0 seq_num 1772) 2014-08-08T10:50:18.961 INFO:tasks.rados.rados.0.vpm137.stdout:3593: done (5 left) 2014-08-08T10:50:18.961 INFO:tasks.rados.rados.0.vpm137.stdout:3594: done (4 left) 2014-08-08T10:50:18.962 INFO:tasks.rados.rados.0.vpm137.stdout:3595: write oid 253 current snap is 0 2014-08-08T10:50:18.962 INFO:tasks.rados.rados.0.vpm137.stdout:3595: seq_num 2206 ranges {587780=718941,2075657=512123} 2014-08-08T10:50:18.988 INFO:tasks.rados.rados.0.vpm137.stdout:3595: writing vpm1375878-253 from 587780 to 1306721 tid 1 2014-08-08T10:50:19.016 INFO:tasks.rados.rados.0.vpm137.stdout:3595: writing vpm1375878-253 from 2075657 to 2587780 tid 2 2014-08-08T10:50:19.018 INFO:tasks.rados.rados.0.vpm137.stdout:3589: finishing write tid 1 to vpm1375878-52 2014-08-08T10:50:19.018 INFO:tasks.rados.rados.0.vpm137.stdout:3589: finishing write tid 2 to vpm1375878-52 2014-08-08T10:50:19.019 INFO:tasks.rados.rados.0.vpm137.stdout:3589: finishing write tid 4 to vpm1375878-52 2014-08-08T10:50:19.019 INFO:tasks.rados.rados.0.vpm137.stdout:3591: finishing write tid 1 to vpm1375878-360 2014-08-08T10:50:19.020 INFO:tasks.rados.rados.0.vpm137.stdout:3591: finishing write tid 2 to vpm1375878-360 2014-08-08T10:50:19.020 INFO:tasks.rados.rados.0.vpm137.stdout:3591: finishing write tid 3 to vpm1375878-360 2014-08-08T10:50:19.020 INFO:tasks.rados.rados.0.vpm137.stdout:3591: finishing write tid 5 to vpm1375878-360 2014-08-08T10:50:19.020 INFO:tasks.rados.rados.0.vpm137.stdout:3591: finishing write tid 6 to vpm1375878-360 2014-08-08T10:50:19.020 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 360 v 383 (ObjNum 2204 snap 0 seq_num 2204) dirty exists 2014-08-08T10:50:19.021 INFO:tasks.rados.rados.0.vpm137.stdout:3589: finishing write tid 5 to vpm1375878-52 2014-08-08T10:50:19.021 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 52 v 349 (ObjNum 2203 snap 0 seq_num 2203) dirty exists 2014-08-08T10:50:19.021 INFO:tasks.rados.rados.0.vpm137.stdout:3592: expect (ObjNum 669 snap 0 seq_num 669) 2014-08-08T10:50:19.343 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 836569748, u'bytes_used': 1766649856, u'op_per_sec': 6, u'version': 1673, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 719859, u'write_bytes_sec': 711673, u'bytes_avail': 427520364544}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:19.344 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:19.538 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:19.538 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:19.733 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:21.184 INFO:tasks.rados.rados.0.vpm137.stdout:3589: done (4 left) 2014-08-08T10:50:21.184 INFO:tasks.rados.rados.0.vpm137.stdout:3591: done (3 left) 2014-08-08T10:50:21.184 INFO:tasks.rados.rados.0.vpm137.stdout:3592: done (2 left) 2014-08-08T10:50:21.185 INFO:tasks.rados.rados.0.vpm137.stdout:3596: write oid 52 current snap is 0 2014-08-08T10:50:21.185 INFO:tasks.rados.rados.0.vpm137.stdout:3596: seq_num 2207 ranges {381185=1} 2014-08-08T10:50:21.187 INFO:tasks.rados.rados.0.vpm137.stdout:3596: writing vpm1375878-52 from 381185 to 381186 tid 1 2014-08-08T10:50:21.188 INFO:tasks.rados.rados.0.vpm137.stdout:3597: read oid 385 snap -1 2014-08-08T10:50:21.189 INFO:tasks.rados.rados.0.vpm137.stdout:3590: expect (ObjNum 1920 snap 0 seq_num 1920) 2014-08-08T10:50:22.763 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:22.964 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:23.007 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:23.079 INFO:tasks.rados.rados.0.vpm137.stdout:3590: done (3 left) 2014-08-08T10:50:23.080 INFO:tasks.rados.rados.0.vpm137.stdout:3598: write oid 497 current snap is 0 2014-08-08T10:50:23.080 INFO:tasks.rados.rados.0.vpm137.stdout:3598: seq_num 2208 ranges {76414=1} 2014-08-08T10:50:23.081 INFO:tasks.rados.rados.0.vpm137.stdout:3598: writing vpm1375878-497 from 76414 to 76415 tid 1 2014-08-08T10:50:23.082 INFO:tasks.rados.rados.0.vpm137.stdout:3599: read oid 69 snap -1 2014-08-08T10:50:23.083 INFO:tasks.rados.rados.0.vpm137.stdout:3600: read oid 381 snap -1 2014-08-08T10:50:23.083 INFO:tasks.rados.rados.0.vpm137.stdout:3595: finishing write tid 1 to vpm1375878-253 2014-08-08T10:50:23.084 INFO:tasks.rados.rados.0.vpm137.stdout:3595: finishing write tid 2 to vpm1375878-253 2014-08-08T10:50:23.084 INFO:tasks.rados.rados.0.vpm137.stdout:3595: finishing write tid 4 to vpm1375878-253 2014-08-08T10:50:23.084 INFO:tasks.rados.rados.0.vpm137.stdout:3595: finishing write tid 5 to vpm1375878-253 2014-08-08T10:50:23.084 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 253 v 434 (ObjNum 2206 snap 0 seq_num 2206) dirty exists 2014-08-08T10:50:23.085 INFO:tasks.rados.rados.0.vpm137.stdout:3597: expect (ObjNum 1524 snap 0 seq_num 1524) 2014-08-08T10:50:24.309 INFO:tasks.rados.rados.0.vpm137.stdout:3595: done (5 left) 2014-08-08T10:50:24.309 INFO:tasks.rados.rados.0.vpm137.stdout:3597: done (4 left) 2014-08-08T10:50:24.310 INFO:tasks.rados.rados.0.vpm137.stdout:3601: write oid 156 current snap is 0 2014-08-08T10:50:24.310 INFO:tasks.rados.rados.0.vpm137.stdout:3601: seq_num 2209 ranges {289020=1} 2014-08-08T10:50:24.312 INFO:tasks.rados.rados.0.vpm137.stdout:3601: writing vpm1375878-156 from 289020 to 289021 tid 1 2014-08-08T10:50:24.313 INFO:tasks.rados.rados.0.vpm137.stdout:3602: write oid 229 current snap is 0 2014-08-08T10:50:24.313 INFO:tasks.rados.rados.0.vpm137.stdout:3602: seq_num 2210 ranges {481403=642254,1606745=468242,2081402=1} 2014-08-08T10:50:24.344 INFO:tasks.rados.rados.0.vpm137.stdout:3602: writing vpm1375878-229 from 481403 to 1123657 tid 1 2014-08-08T10:50:24.373 INFO:tasks.rados.rados.0.vpm137.stdout:3602: writing vpm1375878-229 from 1606745 to 2074987 tid 2 2014-08-08T10:50:24.375 INFO:tasks.rados.rados.0.vpm137.stdout:3602: writing vpm1375878-229 from 2081402 to 2081403 tid 3 2014-08-08T10:50:24.376 INFO:tasks.rados.rados.0.vpm137.stdout:3596: finishing write tid 1 to vpm1375878-52 2014-08-08T10:50:24.377 INFO:tasks.rados.rados.0.vpm137.stdout:3596: finishing write tid 3 to vpm1375878-52 2014-08-08T10:50:24.377 INFO:tasks.rados.rados.0.vpm137.stdout:3596: finishing write tid 4 to vpm1375878-52 2014-08-08T10:50:24.377 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 52 v 351 (ObjNum 2207 snap 0 seq_num 2207) dirty exists 2014-08-08T10:50:24.378 INFO:tasks.rados.rados.0.vpm137.stdout:3600: expect (ObjNum 1831 snap 0 seq_num 1831) 2014-08-08T10:50:24.384 INFO:tasks.rados.rados.0.vpm137.stdout:3596: done (5 left) 2014-08-08T10:50:24.385 INFO:tasks.rados.rados.0.vpm137.stdout:3600: done (4 left) 2014-08-08T10:50:24.385 INFO:tasks.rados.rados.0.vpm137.stdout:3603: write oid 390 current snap is 0 2014-08-08T10:50:24.385 INFO:tasks.rados.rados.0.vpm137.stdout:3603: seq_num 2211 ranges {576632=765482,1776631=1} 2014-08-08T10:50:24.414 INFO:tasks.rados.rados.0.vpm137.stdout:3603: writing vpm1375878-390 from 576632 to 1342114 tid 1 2014-08-08T10:50:24.422 INFO:tasks.rados.rados.0.vpm137.stdout:3603: writing vpm1375878-390 from 1776631 to 1776632 tid 2 2014-08-08T10:50:24.423 INFO:tasks.rados.rados.0.vpm137.stdout:3598: finishing write tid 1 to vpm1375878-497 2014-08-08T10:50:24.423 INFO:tasks.rados.rados.0.vpm137.stdout:3598: finishing write tid 3 to vpm1375878-497 2014-08-08T10:50:24.423 INFO:tasks.rados.rados.0.vpm137.stdout:3598: finishing write tid 4 to vpm1375878-497 2014-08-08T10:50:24.423 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 497 v 353 (ObjNum 2208 snap 0 seq_num 2208) dirty exists 2014-08-08T10:50:24.424 INFO:tasks.rados.rados.0.vpm137.stdout:3599: expect (ObjNum 1868 snap 0 seq_num 1868) 2014-08-08T10:50:25.203 INFO:tasks.rados.rados.0.vpm137.stdout:3598: done (4 left) 2014-08-08T10:50:25.204 INFO:tasks.rados.rados.0.vpm137.stdout:3599: done (3 left) 2014-08-08T10:50:25.204 INFO:tasks.rados.rados.0.vpm137.stdout:3604: write oid 141 current snap is 0 2014-08-08T10:50:25.205 INFO:tasks.rados.rados.0.vpm137.stdout:3604: seq_num 2212 ranges {672886=620935,1472885=1} 2014-08-08T10:50:25.233 INFO:tasks.rados.rados.0.vpm137.stdout:3604: writing vpm1375878-141 from 672886 to 1293821 tid 1 2014-08-08T10:50:25.241 INFO:tasks.rados.rados.0.vpm137.stdout:3604: writing vpm1375878-141 from 1472885 to 1472886 tid 2 2014-08-08T10:50:25.241 INFO:tasks.rados.rados.0.vpm137.stdout:3601: finishing write tid 1 to vpm1375878-156 2014-08-08T10:50:25.242 INFO:tasks.rados.rados.0.vpm137.stdout:3601: finishing write tid 3 to vpm1375878-156 2014-08-08T10:50:25.242 INFO:tasks.rados.rados.0.vpm137.stdout:3601: finishing write tid 4 to vpm1375878-156 2014-08-08T10:50:25.242 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 156 v 599 (ObjNum 2209 snap 0 seq_num 2209) dirty exists 2014-08-08T10:50:25.243 INFO:tasks.rados.rados.0.vpm137.stdout:3602: finishing write tid 1 to vpm1375878-229 2014-08-08T10:50:25.243 INFO:tasks.rados.rados.0.vpm137.stdout:3602: finishing write tid 2 to vpm1375878-229 2014-08-08T10:50:25.244 INFO:tasks.rados.rados.0.vpm137.stdout:3602: finishing write tid 3 to vpm1375878-229 2014-08-08T10:50:25.244 INFO:tasks.rados.rados.0.vpm137.stdout:3602: finishing write tid 5 to vpm1375878-229 2014-08-08T10:50:25.244 INFO:tasks.rados.rados.0.vpm137.stdout:3603: finishing write tid 1 to vpm1375878-390 2014-08-08T10:50:25.244 INFO:tasks.rados.rados.0.vpm137.stdout:3603: finishing write tid 2 to vpm1375878-390 2014-08-08T10:50:25.244 INFO:tasks.rados.rados.0.vpm137.stdout:3603: finishing write tid 4 to vpm1375878-390 2014-08-08T10:50:25.245 INFO:tasks.rados.rados.0.vpm137.stdout:3602: finishing write tid 6 to vpm1375878-229 2014-08-08T10:50:25.245 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 229 v 393 (ObjNum 2210 snap 0 seq_num 2210) dirty exists 2014-08-08T10:50:25.245 INFO:tasks.rados.rados.0.vpm137.stdout:3603: finishing write tid 5 to vpm1375878-390 2014-08-08T10:50:25.245 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 390 v 437 (ObjNum 2211 snap 0 seq_num 2211) dirty exists 2014-08-08T10:50:25.246 INFO:tasks.rados.rados.0.vpm137.stdout:3601: done (3 left) 2014-08-08T10:50:25.246 INFO:tasks.rados.rados.0.vpm137.stdout:3602: done (2 left) 2014-08-08T10:50:25.246 INFO:tasks.rados.rados.0.vpm137.stdout:3603: done (1 left) 2014-08-08T10:50:25.246 INFO:tasks.rados.rados.0.vpm137.stdout:3605: read oid 224 snap -1 2014-08-08T10:50:25.247 INFO:tasks.rados.rados.0.vpm137.stdout:3606: write oid 458 current snap is 0 2014-08-08T10:50:25.247 INFO:tasks.rados.rados.0.vpm137.stdout:3606: seq_num 2213 ranges {465268=745187,1796693=429228,2742184=520611,3265267=1} 2014-08-08T10:50:25.273 INFO:tasks.rados.rados.0.vpm137.stdout:3606: writing vpm1375878-458 from 465268 to 1210455 tid 1 2014-08-08T10:50:25.297 INFO:tasks.rados.rados.0.vpm137.stdout:3606: writing vpm1375878-458 from 1796693 to 2225921 tid 2 2014-08-08T10:50:25.325 INFO:tasks.rados.rados.0.vpm137.stdout:3606: writing vpm1375878-458 from 2742184 to 3262795 tid 3 2014-08-08T10:50:25.327 INFO:tasks.rados.rados.0.vpm137.stdout:3606: writing vpm1375878-458 from 3265267 to 3265268 tid 4 2014-08-08T10:50:25.327 INFO:tasks.rados.rados.0.vpm137.stdout:3604: finishing write tid 1 to vpm1375878-141 2014-08-08T10:50:25.328 INFO:tasks.rados.rados.0.vpm137.stdout:3604: finishing write tid 2 to vpm1375878-141 2014-08-08T10:50:25.328 INFO:tasks.rados.rados.0.vpm137.stdout:3604: finishing write tid 4 to vpm1375878-141 2014-08-08T10:50:25.328 INFO:tasks.rados.rados.0.vpm137.stdout:3604: finishing write tid 5 to vpm1375878-141 2014-08-08T10:50:25.328 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 141 v 386 (ObjNum 2212 snap 0 seq_num 2212) dirty exists 2014-08-08T10:50:25.329 INFO:tasks.rados.rados.0.vpm137.stdout:3605: expect (ObjNum 2096 snap 0 seq_num 2096) 2014-08-08T10:50:25.945 INFO:tasks.rados.rados.0.vpm137.stdout:3604: done (2 left) 2014-08-08T10:50:25.946 INFO:tasks.rados.rados.0.vpm137.stdout:3605: done (1 left) 2014-08-08T10:50:25.946 INFO:tasks.rados.rados.0.vpm137.stdout:3607: write oid 202 current snap is 0 2014-08-08T10:50:25.946 INFO:tasks.rados.rados.0.vpm137.stdout:3607: seq_num 2214 ranges {677874=716991,2027177=747999,3191985=285889} 2014-08-08T10:50:25.974 INFO:tasks.rados.rados.0.vpm137.stdout:3607: writing vpm1375878-202 from 677874 to 1394865 tid 1 2014-08-08T10:50:26.006 INFO:tasks.rados.rados.0.vpm137.stdout:3607: writing vpm1375878-202 from 2027177 to 2775176 tid 2 2014-08-08T10:50:26.022 INFO:tasks.rados.rados.0.vpm137.stdout:3607: writing vpm1375878-202 from 3191985 to 3477874 tid 3 2014-08-08T10:50:26.025 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 1 to vpm1375878-458 2014-08-08T10:50:26.025 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 2 to vpm1375878-458 2014-08-08T10:50:26.025 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 3 to vpm1375878-458 2014-08-08T10:50:26.025 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 4 to vpm1375878-458 2014-08-08T10:50:26.026 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 6 to vpm1375878-458 2014-08-08T10:50:26.026 INFO:tasks.rados.rados.0.vpm137.stdout:3606: finishing write tid 7 to vpm1375878-458 2014-08-08T10:50:26.026 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 458 v 299 (ObjNum 2213 snap 0 seq_num 2213) dirty exists 2014-08-08T10:50:26.026 INFO:tasks.rados.rados.0.vpm137.stdout:3607: finishing write tid 1 to vpm1375878-202 2014-08-08T10:50:26.027 INFO:tasks.rados.rados.0.vpm137.stdout:3606: done (1 left) 2014-08-08T10:50:26.027 INFO:tasks.rados.rados.0.vpm137.stdout:3608: write oid 51 current snap is 0 2014-08-08T10:50:26.027 INFO:tasks.rados.rados.0.vpm137.stdout:3608: seq_num 2215 ranges {774127=440220,2009085=453266,2896058=278069} 2014-08-08T10:50:26.045 INFO:tasks.rados.rados.0.vpm137.stdout:3608: writing vpm1375878-51 from 774127 to 1214347 tid 1 2014-08-08T10:50:26.067 INFO:tasks.rados.rados.0.vpm137.stdout:3608: writing vpm1375878-51 from 2009085 to 2462351 tid 2 2014-08-08T10:50:26.081 INFO:tasks.rados.rados.0.vpm137.stdout:3608: writing vpm1375878-51 from 2896058 to 3174127 tid 3 2014-08-08T10:50:26.081 INFO:tasks.rados.rados.0.vpm137.stdout:3607: finishing write tid 2 to vpm1375878-202 2014-08-08T10:50:26.082 INFO:tasks.rados.rados.0.vpm137.stdout:3607: finishing write tid 3 to vpm1375878-202 2014-08-08T10:50:26.082 INFO:tasks.rados.rados.0.vpm137.stdout:3607: finishing write tid 5 to vpm1375878-202 2014-08-08T10:50:26.082 INFO:tasks.rados.rados.0.vpm137.stdout:3609: write oid 62 current snap is 0 2014-08-08T10:50:26.082 INFO:tasks.rados.rados.0.vpm137.stdout:3609: seq_num 2216 ranges {566509=400000} 2014-08-08T10:50:26.099 INFO:tasks.rados.rados.0.vpm137.stdout:3609: writing vpm1375878-62 from 566509 to 966509 tid 1 2014-08-08T10:50:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:3607: finishing write tid 6 to vpm1375878-202 2014-08-08T10:50:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 202 v 432 (ObjNum 2214 snap 0 seq_num 2214) dirty exists 2014-08-08T10:50:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:3608: finishing write tid 1 to vpm1375878-51 2014-08-08T10:50:26.101 INFO:tasks.rados.rados.0.vpm137.stdout:3607: done (2 left) 2014-08-08T10:50:26.102 INFO:tasks.rados.rados.0.vpm137.stdout:3610: read oid 365 snap -1 2014-08-08T10:50:26.102 INFO:tasks.rados.rados.0.vpm137.stdout:3611: read oid 86 snap -1 2014-08-08T10:50:26.102 INFO:tasks.rados.rados.0.vpm137.stdout:3612: read oid 171 snap -1 2014-08-08T10:50:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3613: read oid 296 snap -1 2014-08-08T10:50:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3614: write oid 166 current snap is 0 2014-08-08T10:50:26.103 INFO:tasks.rados.rados.0.vpm137.stdout:3614: seq_num 2217 ranges {662762=1} 2014-08-08T10:50:26.107 INFO:tasks.rados.rados.0.vpm137.stdout:3614: writing vpm1375878-166 from 662762 to 662763 tid 1 2014-08-08T10:50:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3615: write oid 220 current snap is 0 2014-08-08T10:50:26.108 INFO:tasks.rados.rados.0.vpm137.stdout:3615: seq_num 2218 ranges {473321=400000} 2014-08-08T10:50:26.142 INFO:tasks.rados.rados.0.vpm137.stdout:3615: writing vpm1375878-220 from 473321 to 873321 tid 1 2014-08-08T10:50:26.143 INFO:tasks.rados.rados.0.vpm137.stdout:3616: read oid 454 snap -1 2014-08-08T10:50:26.144 INFO:tasks.rados.rados.0.vpm137.stdout:3617: read oid 176 snap -1 2014-08-08T10:50:26.144 INFO:tasks.rados.rados.0.vpm137.stdout:3618: write oid 406 current snap is 0 2014-08-08T10:50:26.144 INFO:tasks.rados.rados.0.vpm137.stdout:3618: seq_num 2219 ranges {418150=514957,1476046=142104} 2014-08-08T10:50:26.166 INFO:tasks.rados.rados.0.vpm137.stdout:3618: writing vpm1375878-406 from 418150 to 933107 tid 1 2014-08-08T10:50:26.176 INFO:tasks.rados.rados.0.vpm137.stdout:3618: writing vpm1375878-406 from 1476046 to 1618150 tid 2 2014-08-08T10:50:26.177 INFO:tasks.rados.rados.0.vpm137.stdout:3608: finishing write tid 2 to vpm1375878-51 2014-08-08T10:50:26.177 INFO:tasks.rados.rados.0.vpm137.stdout:3608: finishing write tid 3 to vpm1375878-51 2014-08-08T10:50:26.177 INFO:tasks.rados.rados.0.vpm137.stdout:3608: finishing write tid 5 to vpm1375878-51 2014-08-08T10:50:26.177 INFO:tasks.rados.rados.0.vpm137.stdout:3608: finishing write tid 6 to vpm1375878-51 2014-08-08T10:50:26.178 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 51 v 303 (ObjNum 2215 snap 0 seq_num 2215) dirty exists 2014-08-08T10:50:26.178 INFO:tasks.rados.rados.0.vpm137.stdout:3608: done (10 left) 2014-08-08T10:50:26.178 INFO:tasks.rados.rados.0.vpm137.stdout:3619: write oid 229 current snap is 0 2014-08-08T10:50:26.178 INFO:tasks.rados.rados.0.vpm137.stdout:3619: seq_num 2220 ranges {762980=771433,2088354=274626} 2014-08-08T10:50:26.207 INFO:tasks.rados.rados.0.vpm137.stdout:3619: writing vpm1375878-229 from 762980 to 1534413 tid 1 2014-08-08T10:50:26.221 INFO:tasks.rados.rados.0.vpm137.stdout:3619: writing vpm1375878-229 from 2088354 to 2362980 tid 2 2014-08-08T10:50:26.222 INFO:tasks.rados.rados.0.vpm137.stdout:3620: read oid 448 snap -1 2014-08-08T10:50:26.222 INFO:tasks.rados.rados.0.vpm137.stdout:3621: read oid 195 snap -1 2014-08-08T10:50:26.222 INFO:tasks.rados.rados.0.vpm137.stdout:3622: write oid 38 current snap is 0 2014-08-08T10:50:26.223 INFO:tasks.rados.rados.0.vpm137.stdout:3622: seq_num 2221 ranges {458210=743238,1801463=256747} 2014-08-08T10:50:26.253 INFO:tasks.rados.rados.0.vpm137.stdout:3622: writing vpm1375878-38 from 458210 to 1201448 tid 1 2014-08-08T10:50:26.277 INFO:tasks.rados.rados.0.vpm137.stdout:3622: writing vpm1375878-38 from 1801463 to 2058210 tid 2 2014-08-08T10:50:26.280 INFO:tasks.rados.rados.0.vpm137.stdout:3609: finishing write tid 1 to vpm1375878-62 2014-08-08T10:50:26.280 INFO:tasks.rados.rados.0.vpm137.stdout:3609: finishing write tid 3 to vpm1375878-62 2014-08-08T10:50:26.280 INFO:tasks.rados.rados.0.vpm137.stdout:3609: finishing write tid 4 to vpm1375878-62 2014-08-08T10:50:26.282 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 62 v 341 (ObjNum 2216 snap 0 seq_num 2216) dirty exists 2014-08-08T10:50:26.282 INFO:tasks.rados.rados.0.vpm137.stdout:3613: expect (ObjNum 1158 snap 0 seq_num 1158) 2014-08-08T10:50:26.380 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 829324703, u'bytes_used': 1761492992, u'op_per_sec': 10, u'version': 1675, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1888277, u'write_bytes_sec': 733676, u'bytes_avail': 427525521408}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:26.380 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:26.576 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:26.576 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:26.827 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:27.227 INFO:tasks.rados.rados.0.vpm137.stdout:3609: done (13 left) 2014-08-08T10:50:27.227 INFO:tasks.rados.rados.0.vpm137.stdout:3612: done (12 left) 2014-08-08T10:50:27.227 INFO:tasks.rados.rados.0.vpm137.stdout:3613: done (11 left) 2014-08-08T10:50:27.228 INFO:tasks.rados.rados.0.vpm137.stdout:3623: delete oid 247 current snap is 0 2014-08-08T10:50:27.228 INFO:tasks.rados.rados.0.vpm137.stdout:3614: finishing write tid 1 to vpm1375878-166 2014-08-08T10:50:27.228 INFO:tasks.rados.rados.0.vpm137.stdout:3616: expect (ObjNum 2087 snap 0 seq_num 2087) 2014-08-08T10:50:27.652 INFO:tasks.rados.rados.0.vpm137.stdout:3614: finishing write tid 3 to vpm1375878-166 2014-08-08T10:50:27.652 INFO:tasks.rados.rados.0.vpm137.stdout:3614: finishing write tid 4 to vpm1375878-166 2014-08-08T10:50:27.653 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 166 v 439 (ObjNum 2217 snap 0 seq_num 2217) dirty exists 2014-08-08T10:50:27.653 INFO:tasks.rados.rados.0.vpm137.stdout:3618: finishing write tid 1 to vpm1375878-406 2014-08-08T10:50:27.653 INFO:tasks.rados.rados.0.vpm137.stdout:3618: finishing write tid 2 to vpm1375878-406 2014-08-08T10:50:27.653 INFO:tasks.rados.rados.0.vpm137.stdout:3618: finishing write tid 4 to vpm1375878-406 2014-08-08T10:50:27.654 INFO:tasks.rados.rados.0.vpm137.stdout:3620: expect (ObjNum 1928 snap 0 seq_num 1928) 2014-08-08T10:50:28.684 INFO:tasks.rados.rados.0.vpm137.stdout:3614: done (11 left) 2014-08-08T10:50:28.684 INFO:tasks.rados.rados.0.vpm137.stdout:3616: done (10 left) 2014-08-08T10:50:28.684 INFO:tasks.rados.rados.0.vpm137.stdout:3620: done (9 left) 2014-08-08T10:50:28.684 INFO:tasks.rados.rados.0.vpm137.stdout:3623: done (8 left) 2014-08-08T10:50:28.685 INFO:tasks.rados.rados.0.vpm137.stdout:3624: write oid 198 current snap is 0 2014-08-08T10:50:28.685 INFO:tasks.rados.rados.0.vpm137.stdout:3624: seq_num 2223 ranges {63196=1} 2014-08-08T10:50:28.686 INFO:tasks.rados.rados.0.vpm137.stdout:3624: writing vpm1375878-198 from 63196 to 63197 tid 1 2014-08-08T10:50:28.686 INFO:tasks.rados.rados.0.vpm137.stdout:3625: read oid 26 snap -1 2014-08-08T10:50:28.687 INFO:tasks.rados.rados.0.vpm137.stdout:3626: read oid 74 snap -1 2014-08-08T10:50:28.687 INFO:tasks.rados.rados.0.vpm137.stdout:3627: write oid 79 current snap is 0 2014-08-08T10:50:28.688 INFO:tasks.rados.rados.0.vpm137.stdout:3627: seq_num 2224 ranges {559451=446171,1708787=774367,3221136=538315} 2014-08-08T10:50:28.719 INFO:tasks.rados.rados.0.vpm137.stdout:3627: writing vpm1375878-79 from 559451 to 1005622 tid 1 2014-08-08T10:50:28.777 INFO:tasks.rados.rados.0.vpm137.stdout:3627: writing vpm1375878-79 from 1708787 to 2483154 tid 2 2014-08-08T10:50:28.815 INFO:tasks.rados.rados.0.vpm137.stdout:3627: writing vpm1375878-79 from 3221136 to 3759451 tid 3 2014-08-08T10:50:28.819 INFO:tasks.rados.rados.0.vpm137.stdout:3619: finishing write tid 1 to vpm1375878-229 2014-08-08T10:50:28.820 INFO:tasks.rados.rados.0.vpm137.stdout:3619: finishing write tid 2 to vpm1375878-229 2014-08-08T10:50:28.821 INFO:tasks.rados.rados.0.vpm137.stdout:3619: finishing write tid 4 to vpm1375878-229 2014-08-08T10:50:28.821 INFO:tasks.rados.rados.0.vpm137.stdout:3622: finishing write tid 1 to vpm1375878-38 2014-08-08T10:50:28.823 INFO:tasks.rados.rados.0.vpm137.stdout:3610: expect (ObjNum 1310 snap 0 seq_num 1310) 2014-08-08T10:50:29.520 INFO:tasks.rados.rados.0.vpm137.stdout:3610: done (11 left) 2014-08-08T10:50:29.521 INFO:tasks.rados.rados.0.vpm137.stdout:3628: write oid 446 current snap is 0 2014-08-08T10:50:29.521 INFO:tasks.rados.rados.0.vpm137.stdout:3628: seq_num 2225 ranges {504280=1} 2014-08-08T10:50:29.526 INFO:tasks.rados.rados.0.vpm137.stdout:3628: writing vpm1375878-446 from 504280 to 504281 tid 1 2014-08-08T10:50:29.527 INFO:tasks.rados.rados.0.vpm137.stdout:3622: finishing write tid 2 to vpm1375878-38 2014-08-08T10:50:29.527 INFO:tasks.rados.rados.0.vpm137.stdout:3622: finishing write tid 4 to vpm1375878-38 2014-08-08T10:50:29.528 INFO:tasks.rados.rados.0.vpm137.stdout:3611: expect (ObjNum 2148 snap 0 seq_num 2148) 2014-08-08T10:50:29.854 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:31.120 INFO:tasks.rados.rados.0.vpm137.stdout:3611: done (11 left) 2014-08-08T10:50:31.120 INFO:tasks.rados.rados.0.vpm137.stdout:3629: write oid 159 current snap is 0 2014-08-08T10:50:31.121 INFO:tasks.rados.rados.0.vpm137.stdout:3629: seq_num 2226 ranges {447063=541204,1247062=1} 2014-08-08T10:50:31.164 INFO:tasks.rados.rados.0.vpm137.stdout:3629: writing vpm1375878-159 from 447063 to 988267 tid 1 2014-08-08T10:50:31.174 INFO:tasks.rados.rados.0.vpm137.stdout:3629: writing vpm1375878-159 from 1247062 to 1247063 tid 2 2014-08-08T10:50:31.175 INFO:tasks.rados.rados.0.vpm137.stdout:3617: expect (ObjNum 1242 snap 0 seq_num 1242) 2014-08-08T10:50:32.563 INFO:tasks.rados.rados.0.vpm137.stdout:3617: done (11 left) 2014-08-08T10:50:32.565 INFO:tasks.rados.rados.0.vpm137.stdout:3621: done (10 left) 2014-08-08T10:50:32.565 INFO:tasks.rados.rados.0.vpm137.stdout:3630: write oid 384 current snap is 0 2014-08-08T10:50:32.565 INFO:tasks.rados.rados.0.vpm137.stdout:3630: seq_num 2227 ranges {659668=515056,1459667=1} 2014-08-08T10:50:32.597 INFO:tasks.rados.rados.0.vpm137.stdout:3630: writing vpm1375878-384 from 659668 to 1174724 tid 1 2014-08-08T10:50:32.603 INFO:tasks.rados.rados.0.vpm137.stdout:3630: writing vpm1375878-384 from 1459667 to 1459668 tid 2 2014-08-08T10:50:32.604 INFO:tasks.rados.rados.0.vpm137.stdout:3618: finishing write tid 5 to vpm1375878-406 2014-08-08T10:50:32.604 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 406 v 405 (ObjNum 2219 snap 0 seq_num 2219) dirty exists 2014-08-08T10:50:32.604 INFO:tasks.rados.rados.0.vpm137.stdout:3615: finishing write tid 1 to vpm1375878-220 2014-08-08T10:50:32.605 INFO:tasks.rados.rados.0.vpm137.stdout:3615: finishing write tid 3 to vpm1375878-220 2014-08-08T10:50:32.605 INFO:tasks.rados.rados.0.vpm137.stdout:3619: finishing write tid 5 to vpm1375878-229 2014-08-08T10:50:32.605 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 229 v 396 (ObjNum 2220 snap 0 seq_num 2220) dirty exists 2014-08-08T10:50:32.606 INFO:tasks.rados.rados.0.vpm137.stdout:3622: finishing write tid 5 to vpm1375878-38 2014-08-08T10:50:32.606 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 38 v 304 (ObjNum 2221 snap 0 seq_num 2221) dirty exists 2014-08-08T10:50:32.606 INFO:tasks.rados.rados.0.vpm137.stdout:3615: finishing write tid 4 to vpm1375878-220 2014-08-08T10:50:32.606 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 220 v 428 (ObjNum 2218 snap 0 seq_num 2218) dirty exists 2014-08-08T10:50:32.606 INFO:tasks.rados.rados.0.vpm137.stdout:3625: expect (ObjNum 2039 snap 0 seq_num 2039) 2014-08-08T10:50:33.061 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:33.122 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:33.124 INFO:tasks.rados.rados.0.vpm137.stdout:3615: done (10 left) 2014-08-08T10:50:33.124 INFO:tasks.rados.rados.0.vpm137.stdout:3618: done (9 left) 2014-08-08T10:50:33.125 INFO:tasks.rados.rados.0.vpm137.stdout:3619: done (8 left) 2014-08-08T10:50:33.125 INFO:tasks.rados.rados.0.vpm137.stdout:3622: done (7 left) 2014-08-08T10:50:33.125 INFO:tasks.rados.rados.0.vpm137.stdout:3625: done (6 left) 2014-08-08T10:50:33.125 INFO:tasks.rados.rados.0.vpm137.stdout:3631: read oid 171 snap -1 2014-08-08T10:50:33.125 INFO:tasks.rados.rados.0.vpm137.stdout:3632: write oid 416 current snap is 0 2014-08-08T10:50:33.126 INFO:tasks.rados.rados.0.vpm137.stdout:3632: seq_num 2228 ranges {603474=521932,1693123=510351} 2014-08-08T10:50:33.154 INFO:tasks.rados.rados.0.vpm137.stdout:3632: writing vpm1375878-416 from 603474 to 1125406 tid 1 2014-08-08T10:50:33.186 INFO:tasks.rados.rados.0.vpm137.stdout:3632: writing vpm1375878-416 from 1693123 to 2203474 tid 2 2014-08-08T10:50:33.189 INFO:tasks.rados.rados.0.vpm137.stdout:3626: expect (ObjNum 2105 snap 0 seq_num 2105) 2014-08-08T10:50:33.384 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 833800944, u'bytes_used': 1761492992, u'op_per_sec': 15, u'version': 1676, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2313213, u'write_bytes_sec': 1239089, u'bytes_avail': 427525521408}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:33.385 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:33.582 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:33.582 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:33.797 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:34.145 INFO:tasks.rados.rados.0.vpm137.stdout:3626: done (7 left) 2014-08-08T10:50:34.145 INFO:tasks.rados.rados.0.vpm137.stdout:3633: read oid 111 snap -1 2014-08-08T10:50:34.146 INFO:tasks.rados.rados.0.vpm137.stdout:3634: write oid 251 current snap is 0 2014-08-08T10:50:34.146 INFO:tasks.rados.rados.0.vpm137.stdout:3634: seq_num 2229 ranges {548304=493737,1752984=476767,2670143=278161} 2014-08-08T10:50:34.181 INFO:tasks.rados.rados.0.vpm137.stdout:3634: writing vpm1375878-251 from 548304 to 1042041 tid 1 2014-08-08T10:50:34.208 INFO:tasks.rados.rados.0.vpm137.stdout:3634: writing vpm1375878-251 from 1752984 to 2229751 tid 2 2014-08-08T10:50:34.223 INFO:tasks.rados.rados.0.vpm137.stdout:3634: writing vpm1375878-251 from 2670143 to 2948304 tid 3 2014-08-08T10:50:34.224 INFO:tasks.rados.rados.0.vpm137.stdout:3624: finishing write tid 1 to vpm1375878-198 2014-08-08T10:50:34.225 INFO:tasks.rados.rados.0.vpm137.stdout:3624: finishing write tid 3 to vpm1375878-198 2014-08-08T10:50:34.225 INFO:tasks.rados.rados.0.vpm137.stdout:3624: finishing write tid 4 to vpm1375878-198 2014-08-08T10:50:34.227 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 198 v 346 (ObjNum 2223 snap 0 seq_num 2223) dirty exists 2014-08-08T10:50:34.227 INFO:tasks.rados.rados.0.vpm137.stdout:3627: finishing write tid 1 to vpm1375878-79 2014-08-08T10:50:34.227 INFO:tasks.rados.rados.0.vpm137.stdout:3627: finishing write tid 2 to vpm1375878-79 2014-08-08T10:50:34.228 INFO:tasks.rados.rados.0.vpm137.stdout:3627: finishing write tid 3 to vpm1375878-79 2014-08-08T10:50:34.228 INFO:tasks.rados.rados.0.vpm137.stdout:3627: finishing write tid 5 to vpm1375878-79 2014-08-08T10:50:34.228 INFO:tasks.rados.rados.0.vpm137.stdout:3628: finishing write tid 1 to vpm1375878-446 2014-08-08T10:50:34.229 INFO:tasks.rados.rados.0.vpm137.stdout:3628: finishing write tid 3 to vpm1375878-446 2014-08-08T10:50:34.229 INFO:tasks.rados.rados.0.vpm137.stdout:3627: finishing write tid 6 to vpm1375878-79 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 79 v 307 (ObjNum 2224 snap 0 seq_num 2224) dirty exists 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:3628: finishing write tid 4 to vpm1375878-446 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 446 v 309 (ObjNum 2225 snap 0 seq_num 2225) dirty exists 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:3629: finishing write tid 1 to vpm1375878-159 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:3629: finishing write tid 2 to vpm1375878-159 2014-08-08T10:50:34.243 INFO:tasks.rados.rados.0.vpm137.stdout:3629: finishing write tid 4 to vpm1375878-159 2014-08-08T10:50:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:3629: finishing write tid 5 to vpm1375878-159 2014-08-08T10:50:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 159 v 442 (ObjNum 2226 snap 0 seq_num 2226) dirty exists 2014-08-08T10:50:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:3630: finishing write tid 1 to vpm1375878-384 2014-08-08T10:50:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:3630: finishing write tid 2 to vpm1375878-384 2014-08-08T10:50:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:3630: finishing write tid 4 to vpm1375878-384 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:3630: finishing write tid 5 to vpm1375878-384 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 384 v 419 (ObjNum 2227 snap 0 seq_num 2227) dirty exists 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:3632: finishing write tid 1 to vpm1375878-416 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:3632: finishing write tid 2 to vpm1375878-416 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:3632: finishing write tid 4 to vpm1375878-416 2014-08-08T10:50:34.245 INFO:tasks.rados.rados.0.vpm137.stdout:3632: finishing write tid 5 to vpm1375878-416 2014-08-08T10:50:34.246 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 416 v 307 (ObjNum 2228 snap 0 seq_num 2228) dirty exists 2014-08-08T10:50:34.246 INFO:tasks.rados.rados.0.vpm137.stdout:3633: expect (ObjNum 2170 snap 0 seq_num 2170) 2014-08-08T10:50:34.947 INFO:tasks.rados.rados.0.vpm137.stdout:3624: done (8 left) 2014-08-08T10:50:34.948 INFO:tasks.rados.rados.0.vpm137.stdout:3627: done (7 left) 2014-08-08T10:50:34.948 INFO:tasks.rados.rados.0.vpm137.stdout:3628: done (6 left) 2014-08-08T10:50:34.948 INFO:tasks.rados.rados.0.vpm137.stdout:3629: done (5 left) 2014-08-08T10:50:34.949 INFO:tasks.rados.rados.0.vpm137.stdout:3630: done (4 left) 2014-08-08T10:50:34.949 INFO:tasks.rados.rados.0.vpm137.stdout:3631: done (3 left) 2014-08-08T10:50:34.949 INFO:tasks.rados.rados.0.vpm137.stdout:3632: done (2 left) 2014-08-08T10:50:34.949 INFO:tasks.rados.rados.0.vpm137.stdout:3633: done (1 left) 2014-08-08T10:50:34.950 INFO:tasks.rados.rados.0.vpm137.stdout:3635: read oid 40 snap -1 2014-08-08T10:50:34.950 INFO:tasks.rados.rados.0.vpm137.stdout:3636: read oid 420 snap -1 2014-08-08T10:50:34.950 INFO:tasks.rados.rados.0.vpm137.stdout:3637: delete oid 121 current snap is 0 2014-08-08T10:50:34.951 INFO:tasks.rados.rados.0.vpm137.stdout:3634: finishing write tid 1 to vpm1375878-251 2014-08-08T10:50:34.951 INFO:tasks.rados.rados.0.vpm137.stdout:3634: finishing write tid 2 to vpm1375878-251 2014-08-08T10:50:34.951 INFO:tasks.rados.rados.0.vpm137.stdout:3634: finishing write tid 3 to vpm1375878-251 2014-08-08T10:50:34.951 INFO:tasks.rados.rados.0.vpm137.stdout:3634: finishing write tid 5 to vpm1375878-251 2014-08-08T10:50:34.951 INFO:tasks.rados.rados.0.vpm137.stdout:3634: finishing write tid 6 to vpm1375878-251 2014-08-08T10:50:34.952 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 251 v 357 (ObjNum 2229 snap 0 seq_num 2229) dirty exists 2014-08-08T10:50:34.978 INFO:tasks.rados.rados.0.vpm137.stdout:3635: expect (ObjNum 1893 snap 0 seq_num 1893) 2014-08-08T10:50:36.240 INFO:tasks.rados.rados.0.vpm137.stdout:3634: done (3 left) 2014-08-08T10:50:36.240 INFO:tasks.rados.rados.0.vpm137.stdout:3635: done (2 left) 2014-08-08T10:50:36.240 INFO:tasks.rados.rados.0.vpm137.stdout:3637: done (1 left) 2014-08-08T10:50:36.241 INFO:tasks.rados.rados.0.vpm137.stdout:3638: read oid 240 snap -1 2014-08-08T10:50:36.241 INFO:tasks.rados.rados.0.vpm137.stdout:3639: write oid 227 current snap is 0 2014-08-08T10:50:36.241 INFO:tasks.rados.rados.0.vpm137.stdout:3639: seq_num 2231 ranges {588363=589793,1697599=768581,2940368=447995} 2014-08-08T10:50:36.278 INFO:tasks.rados.rados.0.vpm137.stdout:3639: writing vpm1375878-227 from 588363 to 1178156 tid 1 2014-08-08T10:50:36.349 INFO:tasks.rados.rados.0.vpm137.stdout:3639: writing vpm1375878-227 from 1697599 to 2466180 tid 2 2014-08-08T10:50:36.379 INFO:tasks.rados.rados.0.vpm137.stdout:3639: writing vpm1375878-227 from 2940368 to 3388363 tid 3 2014-08-08T10:50:36.453 INFO:tasks.rados.rados.0.vpm137.stdout:3636: expect (ObjNum 2139 snap 0 seq_num 2139) 2014-08-08T10:50:36.877 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:37.124 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:37.189 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:37.352 INFO:tasks.rados.rados.0.vpm137.stdout:3636: done (2 left) 2014-08-08T10:50:37.353 INFO:tasks.rados.rados.0.vpm137.stdout:3640: read oid 372 snap -1 2014-08-08T10:50:37.354 INFO:tasks.rados.rados.0.vpm137.stdout:3641: read oid 323 snap -1 2014-08-08T10:50:37.354 INFO:tasks.rados.rados.0.vpm137.stdout:3642: delete oid 148 current snap is 0 2014-08-08T10:50:37.354 INFO:tasks.rados.rados.0.vpm137.stdout:3639: finishing write tid 1 to vpm1375878-227 2014-08-08T10:50:37.367 INFO:tasks.rados.rados.0.vpm137.stdout:3639: finishing write tid 2 to vpm1375878-227 2014-08-08T10:50:37.367 INFO:tasks.rados.rados.0.vpm137.stdout:3639: finishing write tid 3 to vpm1375878-227 2014-08-08T10:50:37.367 INFO:tasks.rados.rados.0.vpm137.stdout:3639: finishing write tid 5 to vpm1375878-227 2014-08-08T10:50:37.410 INFO:tasks.rados.rados.0.vpm137.stdout:3640: expect (ObjNum 1944 snap 0 seq_num 1944) 2014-08-08T10:50:37.503 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 833394315, u'bytes_used': 1762799616, u'op_per_sec': 13, u'version': 1677, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2172518, u'write_bytes_sec': 1153360, u'bytes_avail': 427524214784}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:37.504 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:37.759 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:37.760 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:37.968 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:38.861 INFO:tasks.rados.rados.0.vpm137.stdout:3638: done (4 left) 2014-08-08T10:50:38.861 INFO:tasks.rados.rados.0.vpm137.stdout:3640: done (3 left) 2014-08-08T10:50:38.862 INFO:tasks.rados.rados.0.vpm137.stdout:3642: done (2 left) 2014-08-08T10:50:38.862 INFO:tasks.rados.rados.0.vpm137.stdout:3643: read oid 305 snap -1 2014-08-08T10:50:38.862 INFO:tasks.rados.rados.0.vpm137.stdout:3644: write oid 106 current snap is 0 2014-08-08T10:50:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3644: seq_num 2233 ranges {344774=1} 2014-08-08T10:50:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3644: writing vpm1375878-106 from 344774 to 344775 tid 1 2014-08-08T10:50:38.865 INFO:tasks.rados.rados.0.vpm137.stdout:3645: read oid 90 snap -1 2014-08-08T10:50:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3639: finishing write tid 6 to vpm1375878-227 2014-08-08T10:50:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 227 v 400 (ObjNum 2231 snap 0 seq_num 2231) dirty exists 2014-08-08T10:50:38.866 INFO:tasks.rados.rados.0.vpm137.stdout:3641: expect (ObjNum 1656 snap 0 seq_num 1656) 2014-08-08T10:50:40.025 INFO:tasks.rados.rados.0.vpm137.stdout:3639: done (4 left) 2014-08-08T10:50:40.025 INFO:tasks.rados.rados.0.vpm137.stdout:3641: done (3 left) 2014-08-08T10:50:40.025 INFO:tasks.rados.rados.0.vpm137.stdout:3646: read oid 43 snap -1 2014-08-08T10:50:40.025 INFO:tasks.rados.rados.0.vpm137.stdout:3647: read oid 269 snap -1 2014-08-08T10:50:40.026 INFO:tasks.rados.rados.0.vpm137.stdout:3648: write oid 439 current snap is 0 2014-08-08T10:50:40.027 INFO:tasks.rados.rados.0.vpm137.stdout:3648: seq_num 2234 ranges {688580=400000} 2014-08-08T10:50:40.059 INFO:tasks.rados.rados.0.vpm137.stdout:3648: writing vpm1375878-439 from 688580 to 1088580 tid 1 2014-08-08T10:50:40.061 INFO:tasks.rados.rados.0.vpm137.stdout:3644: finishing write tid 1 to vpm1375878-106 2014-08-08T10:50:40.065 INFO:tasks.rados.rados.0.vpm137.stdout:3644: finishing write tid 3 to vpm1375878-106 2014-08-08T10:50:40.065 INFO:tasks.rados.rados.0.vpm137.stdout:3649: read oid 102 snap -1 2014-08-08T10:50:40.066 INFO:tasks.rados.rados.0.vpm137.stdout:3650: read oid 312 snap -1 2014-08-08T10:50:40.066 INFO:tasks.rados.rados.0.vpm137.stdout:3651: write oid 149 current snap is 0 2014-08-08T10:50:40.066 INFO:tasks.rados.rados.0.vpm137.stdout:3651: seq_num 2235 ranges {784833=1} 2014-08-08T10:50:40.072 INFO:tasks.rados.rados.0.vpm137.stdout:3651: writing vpm1375878-149 from 784833 to 784834 tid 1 2014-08-08T10:50:40.073 INFO:tasks.rados.rados.0.vpm137.stdout:3643: expect (ObjNum 1326 snap 0 seq_num 1326) 2014-08-08T10:50:40.399 INFO:tasks.rados.rados.0.vpm137.stdout:3643: done (8 left) 2014-08-08T10:50:40.399 INFO:tasks.rados.rados.0.vpm137.stdout:3652: read oid 267 snap -1 2014-08-08T10:50:40.399 INFO:tasks.rados.rados.0.vpm137.stdout:3653: write oid 187 current snap is 0 2014-08-08T10:50:40.400 INFO:tasks.rados.rados.0.vpm137.stdout:3653: seq_num 2236 ranges {444992=519983,1643621=401371} 2014-08-08T10:50:40.420 INFO:tasks.rados.rados.0.vpm137.stdout:3653: writing vpm1375878-187 from 444992 to 964975 tid 1 2014-08-08T10:50:40.441 INFO:tasks.rados.rados.0.vpm137.stdout:3653: writing vpm1375878-187 from 1643621 to 2044992 tid 2 2014-08-08T10:50:40.443 INFO:tasks.rados.rados.0.vpm137.stdout:3645: expect (ObjNum 89 snap 0 seq_num 89) 2014-08-08T10:50:40.999 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:41.209 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:41.265 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:41.492 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 831082471, u'bytes_used': 1763790848, u'op_per_sec': 9, u'version': 1678, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2327421, u'write_bytes_sec': 555010, u'bytes_avail': 427523223552}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:41.492 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:41.717 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:41.718 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:41.916 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:42.402 INFO:tasks.rados.rados.0.vpm137.stdout:3645: done (9 left) 2014-08-08T10:50:42.403 INFO:tasks.rados.rados.0.vpm137.stdout:3654: write oid 500 current snap is 0 2014-08-08T10:50:42.403 INFO:tasks.rados.rados.0.vpm137.stdout:3654: seq_num 2237 ranges {789822=491787,2006329=575224,2789821=1} 2014-08-08T10:50:42.438 INFO:tasks.rados.rados.0.vpm137.stdout:3654: writing vpm1375878-500 from 789822 to 1281609 tid 1 2014-08-08T10:50:42.466 INFO:tasks.rados.rados.0.vpm137.stdout:3654: writing vpm1375878-500 from 2006329 to 2581553 tid 2 2014-08-08T10:50:42.470 INFO:tasks.rados.rados.0.vpm137.stdout:3654: writing vpm1375878-500 from 2789821 to 2789822 tid 3 2014-08-08T10:50:42.470 INFO:tasks.rados.rados.0.vpm137.stdout:3644: finishing write tid 4 to vpm1375878-106 2014-08-08T10:50:42.470 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 106 v 430 (ObjNum 2233 snap 0 seq_num 2233) dirty exists 2014-08-08T10:50:42.471 INFO:tasks.rados.rados.0.vpm137.stdout:3647: expect (ObjNum 1918 snap 0 seq_num 1918) 2014-08-08T10:50:43.515 INFO:tasks.rados.rados.0.vpm137.stdout:3644: done (9 left) 2014-08-08T10:50:43.516 INFO:tasks.rados.rados.0.vpm137.stdout:3646: done (8 left) 2014-08-08T10:50:43.516 INFO:tasks.rados.rados.0.vpm137.stdout:3647: done (7 left) 2014-08-08T10:50:43.517 INFO:tasks.rados.rados.0.vpm137.stdout:3655: write oid 364 current snap is 0 2014-08-08T10:50:43.518 INFO:tasks.rados.rados.0.vpm137.stdout:3655: seq_num 2238 ranges {485051=616040,1836813=648238} 2014-08-08T10:50:43.546 INFO:tasks.rados.rados.0.vpm137.stdout:3655: writing vpm1375878-364 from 485051 to 1101091 tid 1 2014-08-08T10:50:43.589 INFO:tasks.rados.rados.0.vpm137.stdout:3655: writing vpm1375878-364 from 1836813 to 2485051 tid 2 2014-08-08T10:50:43.592 INFO:tasks.rados.rados.0.vpm137.stdout:3649: expect (ObjNum 2031 snap 0 seq_num 2031) 2014-08-08T10:50:44.198 INFO:tasks.rados.rados.0.vpm137.stdout:3649: done (7 left) 2014-08-08T10:50:44.198 INFO:tasks.rados.rados.0.vpm137.stdout:3656: read oid 11 snap -1 2014-08-08T10:50:44.199 INFO:tasks.rados.rados.0.vpm137.stdout:3657: write oid 258 current snap is 0 2014-08-08T10:50:44.199 INFO:tasks.rados.rados.0.vpm137.stdout:3657: seq_num 2239 ranges {429881=587844,1799521=715615,3229880=1} 2014-08-08T10:50:44.221 INFO:tasks.rados.rados.0.vpm137.stdout:3657: writing vpm1375878-258 from 429881 to 1017725 tid 1 2014-08-08T10:50:44.252 INFO:tasks.rados.rados.0.vpm137.stdout:3657: writing vpm1375878-258 from 1799521 to 2515136 tid 2 2014-08-08T10:50:44.260 INFO:tasks.rados.rados.0.vpm137.stdout:3657: writing vpm1375878-258 from 3229880 to 3229881 tid 3 2014-08-08T10:50:44.261 INFO:tasks.rados.rados.0.vpm137.stdout:3648: finishing write tid 1 to vpm1375878-439 2014-08-08T10:50:44.261 INFO:tasks.rados.rados.0.vpm137.stdout:3648: finishing write tid 3 to vpm1375878-439 2014-08-08T10:50:44.261 INFO:tasks.rados.rados.0.vpm137.stdout:3651: finishing write tid 1 to vpm1375878-149 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:3651: finishing write tid 3 to vpm1375878-149 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:3648: finishing write tid 4 to vpm1375878-439 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 439 v 388 (ObjNum 2234 snap 0 seq_num 2234) dirty exists 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:3651: finishing write tid 4 to vpm1375878-149 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 149 v 432 (ObjNum 2235 snap 0 seq_num 2235) dirty exists 2014-08-08T10:50:44.274 INFO:tasks.rados.rados.0.vpm137.stdout:3650: expect (ObjNum 1879 snap 0 seq_num 1879) 2014-08-08T10:50:44.942 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:45.477 INFO:tasks.rados.rados.0.vpm137.stdout:3648: done (8 left) 2014-08-08T10:50:45.477 INFO:tasks.rados.rados.0.vpm137.stdout:3650: done (7 left) 2014-08-08T10:50:45.477 INFO:tasks.rados.rados.0.vpm137.stdout:3651: done (6 left) 2014-08-08T10:50:45.477 INFO:tasks.rados.rados.0.vpm137.stdout:3658: read oid 170 snap -1 2014-08-08T10:50:45.477 INFO:tasks.rados.rados.0.vpm137.stdout:3659: write oid 459 current snap is 0 2014-08-08T10:50:45.482 INFO:tasks.rados.rados.0.vpm137.stdout:3659: seq_num 2240 ranges {774711=594720,1913653=537234,3229350=745361} 2014-08-08T10:50:45.502 INFO:tasks.rados.rados.0.vpm137.stdout:3659: writing vpm1375878-459 from 774711 to 1369431 tid 1 2014-08-08T10:50:45.533 INFO:tasks.rados.rados.0.vpm137.stdout:3659: writing vpm1375878-459 from 1913653 to 2450887 tid 2 2014-08-08T10:50:45.567 INFO:tasks.rados.rados.0.vpm137.stdout:3659: writing vpm1375878-459 from 3229350 to 3974711 tid 3 2014-08-08T10:50:45.570 INFO:tasks.rados.rados.0.vpm137.stdout:3653: finishing write tid 1 to vpm1375878-187 2014-08-08T10:50:45.570 INFO:tasks.rados.rados.0.vpm137.stdout:3653: finishing write tid 2 to vpm1375878-187 2014-08-08T10:50:45.570 INFO:tasks.rados.rados.0.vpm137.stdout:3653: finishing write tid 4 to vpm1375878-187 2014-08-08T10:50:45.570 INFO:tasks.rados.rados.0.vpm137.stdout:3652: expect (ObjNum 266 snap 0 seq_num 266) 2014-08-08T10:50:46.069 INFO:tasks.rados.rados.0.vpm137.stdout:3652: done (7 left) 2014-08-08T10:50:46.071 INFO:tasks.rados.rados.0.vpm137.stdout:3660: read oid 133 snap -1 2014-08-08T10:50:46.071 INFO:tasks.rados.rados.0.vpm137.stdout:3661: read oid 191 snap -1 2014-08-08T10:50:46.072 INFO:tasks.rados.rados.0.vpm137.stdout:3662: write oid 347 current snap is 0 2014-08-08T10:50:46.072 INFO:tasks.rados.rados.0.vpm137.stdout:3662: seq_num 2241 ranges {186292=1} 2014-08-08T10:50:46.073 INFO:tasks.rados.rados.0.vpm137.stdout:3662: writing vpm1375878-347 from 186292 to 186293 tid 1 2014-08-08T10:50:46.074 INFO:tasks.rados.rados.0.vpm137.stdout:3663: read oid 362 snap -1 2014-08-08T10:50:46.075 INFO:tasks.rados.rados.0.vpm137.stdout:3653: finishing write tid 5 to vpm1375878-187 2014-08-08T10:50:46.075 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 187 v 360 (ObjNum 2236 snap 0 seq_num 2236) dirty exists 2014-08-08T10:50:46.075 INFO:tasks.rados.rados.0.vpm137.stdout:3654: finishing write tid 1 to vpm1375878-500 2014-08-08T10:50:46.075 INFO:tasks.rados.rados.0.vpm137.stdout:3654: finishing write tid 2 to vpm1375878-500 2014-08-08T10:50:46.076 INFO:tasks.rados.rados.0.vpm137.stdout:3654: finishing write tid 3 to vpm1375878-500 2014-08-08T10:50:46.076 INFO:tasks.rados.rados.0.vpm137.stdout:3654: finishing write tid 5 to vpm1375878-500 2014-08-08T10:50:46.077 INFO:tasks.rados.rados.0.vpm137.stdout:3654: finishing write tid 6 to vpm1375878-500 2014-08-08T10:50:46.077 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 500 v 494 (ObjNum 2237 snap 0 seq_num 2237) dirty exists 2014-08-08T10:50:46.080 INFO:tasks.rados.rados.0.vpm137.stdout:3655: finishing write tid 1 to vpm1375878-364 2014-08-08T10:50:46.081 INFO:tasks.rados.rados.0.vpm137.stdout:3655: finishing write tid 2 to vpm1375878-364 2014-08-08T10:50:46.081 INFO:tasks.rados.rados.0.vpm137.stdout:3655: finishing write tid 4 to vpm1375878-364 2014-08-08T10:50:46.082 INFO:tasks.rados.rados.0.vpm137.stdout:3655: finishing write tid 5 to vpm1375878-364 2014-08-08T10:50:46.082 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 364 v 403 (ObjNum 2238 snap 0 seq_num 2238) dirty exists 2014-08-08T10:50:46.082 INFO:tasks.rados.rados.0.vpm137.stdout:3657: finishing write tid 1 to vpm1375878-258 2014-08-08T10:50:46.083 INFO:tasks.rados.rados.0.vpm137.stdout:3657: finishing write tid 2 to vpm1375878-258 2014-08-08T10:50:46.083 INFO:tasks.rados.rados.0.vpm137.stdout:3657: finishing write tid 3 to vpm1375878-258 2014-08-08T10:50:46.083 INFO:tasks.rados.rados.0.vpm137.stdout:3657: finishing write tid 5 to vpm1375878-258 2014-08-08T10:50:46.083 INFO:tasks.rados.rados.0.vpm137.stdout:3657: finishing write tid 6 to vpm1375878-258 2014-08-08T10:50:46.084 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 258 v 392 (ObjNum 2239 snap 0 seq_num 2239) dirty exists 2014-08-08T10:50:46.085 INFO:tasks.rados.rados.0.vpm137.stdout:3659: finishing write tid 1 to vpm1375878-459 2014-08-08T10:50:46.085 INFO:tasks.rados.rados.0.vpm137.stdout:3659: finishing write tid 2 to vpm1375878-459 2014-08-08T10:50:46.085 INFO:tasks.rados.rados.0.vpm137.stdout:3659: finishing write tid 3 to vpm1375878-459 2014-08-08T10:50:46.085 INFO:tasks.rados.rados.0.vpm137.stdout:3659: finishing write tid 5 to vpm1375878-459 2014-08-08T10:50:46.085 INFO:tasks.rados.rados.0.vpm137.stdout:3659: finishing write tid 6 to vpm1375878-459 2014-08-08T10:50:46.086 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 459 v 396 (ObjNum 2240 snap 0 seq_num 2240) dirty exists 2014-08-08T10:50:46.086 INFO:tasks.rados.rados.0.vpm137.stdout:3653: done (10 left) 2014-08-08T10:50:46.087 INFO:tasks.rados.rados.0.vpm137.stdout:3654: done (9 left) 2014-08-08T10:50:46.087 INFO:tasks.rados.rados.0.vpm137.stdout:3655: done (8 left) 2014-08-08T10:50:46.087 INFO:tasks.rados.rados.0.vpm137.stdout:3656: done (7 left) 2014-08-08T10:50:46.087 INFO:tasks.rados.rados.0.vpm137.stdout:3657: done (6 left) 2014-08-08T10:50:46.088 INFO:tasks.rados.rados.0.vpm137.stdout:3658: done (5 left) 2014-08-08T10:50:46.088 INFO:tasks.rados.rados.0.vpm137.stdout:3659: done (4 left) 2014-08-08T10:50:46.088 INFO:tasks.rados.rados.0.vpm137.stdout:3664: write oid 420 current snap is 0 2014-08-08T10:50:46.088 INFO:tasks.rados.rados.0.vpm137.stdout:3664: seq_num 2242 ranges {778674=690777,1953373=25301} 2014-08-08T10:50:46.116 INFO:tasks.rados.rados.0.vpm137.stdout:3664: writing vpm1375878-420 from 778674 to 1469451 tid 1 2014-08-08T10:50:46.124 INFO:tasks.rados.rados.0.vpm137.stdout:3664: writing vpm1375878-420 from 1953373 to 1978674 tid 2 2014-08-08T10:50:46.126 INFO:tasks.rados.rados.0.vpm137.stdout:3660: expect (ObjNum 870 snap 0 seq_num 870) 2014-08-08T10:50:46.314 INFO:tasks.rados.rados.0.vpm137.stdout:3660: done (4 left) 2014-08-08T10:50:46.315 INFO:tasks.rados.rados.0.vpm137.stdout:3665: write oid 37 current snap is 0 2014-08-08T10:50:46.315 INFO:tasks.rados.rados.0.vpm137.stdout:3665: seq_num 2243 ranges {473904=511157,1631409=42495} 2014-08-08T10:50:46.344 INFO:tasks.rados.rados.0.vpm137.stdout:3665: writing vpm1375878-37 from 473904 to 985061 tid 1 2014-08-08T10:50:46.357 INFO:tasks.rados.rados.0.vpm137.stdout:3665: writing vpm1375878-37 from 1631409 to 1673904 tid 2 2014-08-08T10:50:46.358 INFO:tasks.rados.rados.0.vpm137.stdout:3661: expect (ObjNum 1833 snap 0 seq_num 1833) 2014-08-08T10:50:46.466 INFO:tasks.rados.rados.0.vpm137.stdout:3661: done (4 left) 2014-08-08T10:50:46.466 INFO:tasks.rados.rados.0.vpm137.stdout:3666: read oid 436 snap -1 2014-08-08T10:50:46.467 INFO:tasks.rados.rados.0.vpm137.stdout:3667: write oid 303 current snap is 0 2014-08-08T10:50:46.467 INFO:tasks.rados.rados.0.vpm137.stdout:3667: seq_num 2244 ranges {570158=518034,1370157=1} 2014-08-08T10:50:46.519 INFO:tasks.rados.rados.0.vpm137.stdout:3667: writing vpm1375878-303 from 570158 to 1088192 tid 1 2014-08-08T10:50:46.525 INFO:tasks.rados.rados.0.vpm137.stdout:3667: writing vpm1375878-303 from 1370157 to 1370158 tid 2 2014-08-08T10:50:46.527 INFO:tasks.rados.rados.0.vpm137.stdout:3662: finishing write tid 1 to vpm1375878-347 2014-08-08T10:50:46.527 INFO:tasks.rados.rados.0.vpm137.stdout:3662: finishing write tid 3 to vpm1375878-347 2014-08-08T10:50:46.528 INFO:tasks.rados.rados.0.vpm137.stdout:3664: finishing write tid 1 to vpm1375878-420 2014-08-08T10:50:46.529 INFO:tasks.rados.rados.0.vpm137.stdout:3664: finishing write tid 2 to vpm1375878-420 2014-08-08T10:50:46.529 INFO:tasks.rados.rados.0.vpm137.stdout:3664: finishing write tid 4 to vpm1375878-420 2014-08-08T10:50:46.529 INFO:tasks.rados.rados.0.vpm137.stdout:3662: finishing write tid 4 to vpm1375878-347 2014-08-08T10:50:46.530 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 347 v 407 (ObjNum 2241 snap 0 seq_num 2241) dirty exists 2014-08-08T10:50:46.530 INFO:tasks.rados.rados.0.vpm137.stdout:3664: finishing write tid 5 to vpm1375878-420 2014-08-08T10:50:46.531 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 420 v 363 (ObjNum 2242 snap 0 seq_num 2242) dirty exists 2014-08-08T10:50:46.531 INFO:tasks.rados.rados.0.vpm137.stdout:3663: expect (ObjNum 1861 snap 0 seq_num 1861) 2014-08-08T10:50:48.096 INFO:tasks.rados.rados.0.vpm137.stdout:3662: done (5 left) 2014-08-08T10:50:48.097 INFO:tasks.rados.rados.0.vpm137.stdout:3663: done (4 left) 2014-08-08T10:50:48.098 INFO:tasks.rados.rados.0.vpm137.stdout:3664: done (3 left) 2014-08-08T10:50:48.098 INFO:tasks.rados.rados.0.vpm137.stdout:3668: read oid 45 snap -1 2014-08-08T10:50:48.099 INFO:tasks.rados.rados.0.vpm137.stdout:3669: read oid 250 snap -1 2014-08-08T10:50:48.099 INFO:tasks.rados.rados.0.vpm137.stdout:3670: write oid 186 current snap is 0 2014-08-08T10:50:48.099 INFO:tasks.rados.rados.0.vpm137.stdout:3670: seq_num 2245 ranges {631340=490862,1825626=674706,2631339=1} 2014-08-08T10:50:48.125 INFO:tasks.rados.rados.0.vpm137.stdout:3670: writing vpm1375878-186 from 631340 to 1122202 tid 1 2014-08-08T10:50:48.165 INFO:tasks.rados.rados.0.vpm137.stdout:3670: writing vpm1375878-186 from 1825626 to 2500332 tid 2 2014-08-08T10:50:48.167 INFO:tasks.rados.rados.0.vpm137.stdout:3670: writing vpm1375878-186 from 2631339 to 2631340 tid 3 2014-08-08T10:50:48.169 INFO:tasks.rados.rados.0.vpm137.stdout:3665: finishing write tid 1 to vpm1375878-37 2014-08-08T10:50:48.169 INFO:tasks.rados.rados.0.vpm137.stdout:3665: finishing write tid 2 to vpm1375878-37 2014-08-08T10:50:48.169 INFO:tasks.rados.rados.0.vpm137.stdout:3665: finishing write tid 4 to vpm1375878-37 2014-08-08T10:50:48.170 INFO:tasks.rados.rados.0.vpm137.stdout:3665: finishing write tid 5 to vpm1375878-37 2014-08-08T10:50:48.170 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 37 v 435 (ObjNum 2243 snap 0 seq_num 2243) dirty exists 2014-08-08T10:50:48.170 INFO:tasks.rados.rados.0.vpm137.stdout:3666: expect (ObjNum 1511 snap 0 seq_num 1511) 2014-08-08T10:50:48.218 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:48.274 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:48.546 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 833788749, u'bytes_used': 1761898496, u'op_per_sec': 8, u'version': 1679, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1980821, u'write_bytes_sec': 676924, u'bytes_avail': 427525115904}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:50:48.547 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:50.750 INFO:tasks.rados.rados.0.vpm137.stdout:3665: done (5 left) 2014-08-08T10:50:50.750 INFO:tasks.rados.rados.0.vpm137.stdout:3666: done (4 left) 2014-08-08T10:50:50.750 INFO:tasks.rados.rados.0.vpm137.stdout:3671: write oid 189 current snap is 0 2014-08-08T10:50:50.751 INFO:tasks.rados.rados.0.vpm137.stdout:3671: seq_num 2246 ranges {575145=614090,1938733=744901,3181405=193740} 2014-08-08T10:50:50.784 INFO:tasks.rados.rados.0.vpm137.stdout:3671: writing vpm1375878-189 from 575145 to 1189235 tid 1 2014-08-08T10:50:50.837 INFO:tasks.rados.rados.0.vpm137.stdout:3671: writing vpm1375878-189 from 1938733 to 2683634 tid 2 2014-08-08T10:50:50.851 INFO:tasks.rados.rados.0.vpm137.stdout:3671: writing vpm1375878-189 from 3181405 to 3375145 tid 3 2014-08-08T10:50:50.852 INFO:tasks.rados.rados.0.vpm137.stdout:3667: finishing write tid 1 to vpm1375878-303 2014-08-08T10:50:50.853 INFO:tasks.rados.rados.0.vpm137.stdout:3667: finishing write tid 2 to vpm1375878-303 2014-08-08T10:50:50.854 INFO:tasks.rados.rados.0.vpm137.stdout:3667: finishing write tid 4 to vpm1375878-303 2014-08-08T10:50:50.854 INFO:tasks.rados.rados.0.vpm137.stdout:3667: finishing write tid 5 to vpm1375878-303 2014-08-08T10:50:50.854 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 303 v 399 (ObjNum 2244 snap 0 seq_num 2244) dirty exists 2014-08-08T10:50:50.854 INFO:tasks.rados.rados.0.vpm137.stdout:3669: expect (ObjNum 1783 snap 0 seq_num 1783) 2014-08-08T10:50:50.906 INFO:tasks.rados.rados.0.vpm137.stdout:3667: done (4 left) 2014-08-08T10:50:50.907 INFO:tasks.rados.rados.0.vpm137.stdout:3669: done (3 left) 2014-08-08T10:50:50.907 INFO:tasks.rados.rados.0.vpm137.stdout:3672: write oid 360 current snap is 0 2014-08-08T10:50:50.907 INFO:tasks.rados.rados.0.vpm137.stdout:3672: seq_num 2247 ranges {671399=585895,1901442=415096,3071398=1} 2014-08-08T10:50:50.936 INFO:tasks.rados.rados.0.vpm137.stdout:3672: writing vpm1375878-360 from 671399 to 1257294 tid 1 2014-08-08T10:50:50.961 INFO:tasks.rados.rados.0.vpm137.stdout:3672: writing vpm1375878-360 from 1901442 to 2316538 tid 2 2014-08-08T10:50:50.972 INFO:tasks.rados.rados.0.vpm137.stdout:3672: writing vpm1375878-360 from 3071398 to 3071399 tid 3 2014-08-08T10:50:50.972 INFO:tasks.rados.rados.0.vpm137.stdout:3668: expect (ObjNum 665 snap 0 seq_num 665) 2014-08-08T10:50:51.008 INFO:tasks.rados.rados.0.vpm137.stdout:3668: done (3 left) 2014-08-08T10:50:51.008 INFO:tasks.rados.rados.0.vpm137.stdout:3673: write oid 11 current snap is 0 2014-08-08T10:50:51.009 INFO:tasks.rados.rados.0.vpm137.stdout:3673: seq_num 2248 ranges {616229=592771,1615574=636715,2916080=565575,3816228=1} 2014-08-08T10:50:51.031 INFO:tasks.rados.rados.0.vpm137.stdout:3673: writing vpm1375878-11 from 616229 to 1209000 tid 1 2014-08-08T10:50:51.057 INFO:tasks.rados.rados.0.vpm137.stdout:3673: writing vpm1375878-11 from 1615574 to 2252289 tid 2 2014-08-08T10:50:51.093 INFO:tasks.rados.rados.0.vpm137.stdout:3673: writing vpm1375878-11 from 2916080 to 3481655 tid 3 2014-08-08T10:50:51.101 INFO:tasks.rados.rados.0.vpm137.stdout:3673: writing vpm1375878-11 from 3816228 to 3816229 tid 4 2014-08-08T10:50:51.102 INFO:tasks.rados.rados.0.vpm137.stdout:3670: finishing write tid 1 to vpm1375878-186 2014-08-08T10:50:51.102 INFO:tasks.rados.rados.0.vpm137.stdout:3670: finishing write tid 2 to vpm1375878-186 2014-08-08T10:50:51.103 INFO:tasks.rados.rados.0.vpm137.stdout:3670: finishing write tid 3 to vpm1375878-186 2014-08-08T10:50:51.103 INFO:tasks.rados.rados.0.vpm137.stdout:3670: finishing write tid 5 to vpm1375878-186 2014-08-08T10:50:51.103 INFO:tasks.rados.rados.0.vpm137.stdout:3670: finishing write tid 6 to vpm1375878-186 2014-08-08T10:50:51.104 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 186 v 403 (ObjNum 2245 snap 0 seq_num 2245) dirty exists 2014-08-08T10:50:51.104 INFO:tasks.rados.rados.0.vpm137.stdout:3671: finishing write tid 1 to vpm1375878-189 2014-08-08T10:50:51.104 INFO:tasks.rados.rados.0.vpm137.stdout:3671: finishing write tid 2 to vpm1375878-189 2014-08-08T10:50:51.105 INFO:tasks.rados.rados.0.vpm137.stdout:3671: finishing write tid 3 to vpm1375878-189 2014-08-08T10:50:51.105 INFO:tasks.rados.rados.0.vpm137.stdout:3671: finishing write tid 5 to vpm1375878-189 2014-08-08T10:50:51.106 INFO:tasks.rados.rados.0.vpm137.stdout:3672: finishing write tid 1 to vpm1375878-360 2014-08-08T10:50:51.106 INFO:tasks.rados.rados.0.vpm137.stdout:3672: finishing write tid 2 to vpm1375878-360 2014-08-08T10:50:51.106 INFO:tasks.rados.rados.0.vpm137.stdout:3672: finishing write tid 3 to vpm1375878-360 2014-08-08T10:50:51.106 INFO:tasks.rados.rados.0.vpm137.stdout:3672: finishing write tid 5 to vpm1375878-360 2014-08-08T10:50:51.106 INFO:tasks.rados.rados.0.vpm137.stdout:3670: done (3 left) 2014-08-08T10:50:51.107 INFO:tasks.rados.rados.0.vpm137.stdout:3674: write oid 212 current snap is 0 2014-08-08T10:50:51.107 INFO:tasks.rados.rados.0.vpm137.stdout:3674: seq_num 2249 ranges {560033=1} 2014-08-08T10:50:51.108 INFO:tasks.rados.rados.0.vpm137.stdout:3674: writing vpm1375878-212 from 560033 to 560034 tid 1 2014-08-08T10:50:51.109 INFO:tasks.rados.rados.0.vpm137.stdout:3675: write oid 427 current snap is 0 2014-08-08T10:50:51.109 INFO:tasks.rados.rados.0.vpm137.stdout:3675: seq_num 2250 ranges {771615=1} 2014-08-08T10:50:51.116 INFO:tasks.rados.rados.0.vpm137.stdout:3675: writing vpm1375878-427 from 771615 to 771616 tid 1 2014-08-08T10:50:51.116 INFO:tasks.rados.rados.0.vpm137.stdout:3676: write oid 320 current snap is 0 2014-08-08T10:50:51.117 INFO:tasks.rados.rados.0.vpm137.stdout:3676: seq_num 2251 ranges {715422=660632,1515421=1} 2014-08-08T10:50:51.149 INFO:tasks.rados.rados.0.vpm137.stdout:3676: writing vpm1375878-320 from 715422 to 1376054 tid 1 2014-08-08T10:50:51.152 INFO:tasks.rados.rados.0.vpm137.stdout:3676: writing vpm1375878-320 from 1515421 to 1515422 tid 2 2014-08-08T10:50:51.153 INFO:tasks.rados.rados.0.vpm137.stdout:3671: finishing write tid 6 to vpm1375878-189 2014-08-08T10:50:51.153 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 189 v 345 (ObjNum 2246 snap 0 seq_num 2246) dirty exists 2014-08-08T10:50:51.154 INFO:tasks.rados.rados.0.vpm137.stdout:3672: finishing write tid 6 to vpm1375878-360 2014-08-08T10:50:51.154 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 360 v 407 (ObjNum 2247 snap 0 seq_num 2247) dirty exists 2014-08-08T10:50:51.154 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 1 to vpm1375878-11 2014-08-08T10:50:51.155 INFO:tasks.rados.rados.0.vpm137.stdout:3671: done (5 left) 2014-08-08T10:50:51.155 INFO:tasks.rados.rados.0.vpm137.stdout:3672: done (4 left) 2014-08-08T10:50:51.155 INFO:tasks.rados.rados.0.vpm137.stdout:3677: read oid 86 snap -1 2014-08-08T10:50:51.155 INFO:tasks.rados.rados.0.vpm137.stdout:3678: write oid 157 current snap is 0 2014-08-08T10:50:51.155 INFO:tasks.rados.rados.0.vpm137.stdout:3678: seq_num 2252 ranges {660252=668532,1883559=376693} 2014-08-08T10:50:51.186 INFO:tasks.rados.rados.0.vpm137.stdout:3678: writing vpm1375878-157 from 660252 to 1328784 tid 1 2014-08-08T10:50:51.213 INFO:tasks.rados.rados.0.vpm137.stdout:3678: writing vpm1375878-157 from 1883559 to 2260252 tid 2 2014-08-08T10:50:51.213 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 2 to vpm1375878-11 2014-08-08T10:50:51.213 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 3 to vpm1375878-11 2014-08-08T10:50:51.214 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 4 to vpm1375878-11 2014-08-08T10:50:51.214 INFO:tasks.rados.rados.0.vpm137.stdout:3679: read oid 245 snap -1 2014-08-08T10:50:51.214 INFO:tasks.rados.rados.0.vpm137.stdout:3680: write oid 253 current snap is 0 2014-08-08T10:50:51.214 INFO:tasks.rados.rados.0.vpm137.stdout:3680: seq_num 2253 ranges {756505=640337,1956504=1} 2014-08-08T10:50:51.251 INFO:tasks.rados.rados.0.vpm137.stdout:3680: writing vpm1375878-253 from 756505 to 1396842 tid 1 2014-08-08T10:50:51.256 INFO:tasks.rados.rados.0.vpm137.stdout:3680: writing vpm1375878-253 from 1956504 to 1956505 tid 2 2014-08-08T10:50:51.256 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 6 to vpm1375878-11 2014-08-08T10:50:51.257 INFO:tasks.rados.rados.0.vpm137.stdout:3681: write oid 328 current snap is 0 2014-08-08T10:50:51.257 INFO:tasks.rados.rados.0.vpm137.stdout:3681: seq_num 2254 ranges {416663=612141,1640655=443359,2868138=348525} 2014-08-08T10:50:51.295 INFO:tasks.rados.rados.0.vpm137.stdout:3681: writing vpm1375878-328 from 416663 to 1028804 tid 1 2014-08-08T10:50:51.346 INFO:tasks.rados.rados.0.vpm137.stdout:3681: writing vpm1375878-328 from 1640655 to 2084014 tid 2 2014-08-08T10:50:51.373 INFO:tasks.rados.rados.0.vpm137.stdout:3681: writing vpm1375878-328 from 2868138 to 3216663 tid 3 2014-08-08T10:50:51.375 INFO:tasks.rados.rados.0.vpm137.stdout:3674: finishing write tid 1 to vpm1375878-212 2014-08-08T10:50:51.375 INFO:tasks.rados.rados.0.vpm137.stdout:3674: finishing write tid 3 to vpm1375878-212 2014-08-08T10:50:51.375 INFO:tasks.rados.rados.0.vpm137.stdout:3674: finishing write tid 4 to vpm1375878-212 2014-08-08T10:50:51.375 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 212 v 347 (ObjNum 2249 snap 0 seq_num 2249) dirty exists 2014-08-08T10:50:51.376 INFO:tasks.rados.rados.0.vpm137.stdout:3673: finishing write tid 7 to vpm1375878-11 2014-08-08T10:50:51.376 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 11 v 408 (ObjNum 2248 snap 0 seq_num 2248) dirty exists 2014-08-08T10:50:51.376 INFO:tasks.rados.rados.0.vpm137.stdout:3677: expect (ObjNum 2148 snap 0 seq_num 2148) 2014-08-08T10:50:52.652 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:50:52.652 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:52.800 INFO:tasks.rados.rados.0.vpm137.stdout:3673: done (8 left) 2014-08-08T10:50:52.800 INFO:tasks.rados.rados.0.vpm137.stdout:3674: done (7 left) 2014-08-08T10:50:52.800 INFO:tasks.rados.rados.0.vpm137.stdout:3677: done (6 left) 2014-08-08T10:50:52.800 INFO:tasks.rados.rados.0.vpm137.stdout:3682: delete oid 97 current snap is 0 2014-08-08T10:50:52.801 INFO:tasks.rados.rados.0.vpm137.stdout:3675: finishing write tid 1 to vpm1375878-427 2014-08-08T10:50:52.801 INFO:tasks.rados.rados.0.vpm137.stdout:3675: finishing write tid 3 to vpm1375878-427 2014-08-08T10:50:52.801 INFO:tasks.rados.rados.0.vpm137.stdout:3675: finishing write tid 4 to vpm1375878-427 2014-08-08T10:50:52.801 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 427 v 365 (ObjNum 2250 snap 0 seq_num 2250) dirty exists 2014-08-08T10:50:52.802 INFO:tasks.rados.rados.0.vpm137.stdout:3676: finishing write tid 1 to vpm1375878-320 2014-08-08T10:50:52.802 INFO:tasks.rados.rados.0.vpm137.stdout:3679: expect (ObjNum 1916 snap 0 seq_num 1916) 2014-08-08T10:50:53.320 INFO:tasks.rados.rados.0.vpm137.stdout:3675: done (6 left) 2014-08-08T10:50:53.320 INFO:tasks.rados.rados.0.vpm137.stdout:3679: done (5 left) 2014-08-08T10:50:53.321 INFO:tasks.rados.rados.0.vpm137.stdout:3682: done (4 left) 2014-08-08T10:50:53.321 INFO:tasks.rados.rados.0.vpm137.stdout:3683: write oid 144 current snap is 0 2014-08-08T10:50:53.321 INFO:tasks.rados.rados.0.vpm137.stdout:3683: seq_num 2256 ranges {456723=592870,1469944=583749,2471613=780283,3656722=1} 2014-08-08T10:50:53.343 INFO:tasks.rados.rados.0.vpm137.stdout:3683: writing vpm1375878-144 from 456723 to 1049593 tid 1 2014-08-08T10:50:53.380 INFO:tasks.rados.rados.0.vpm137.stdout:3683: writing vpm1375878-144 from 1469944 to 2053693 tid 2 2014-08-08T10:50:53.418 INFO:tasks.rados.rados.0.vpm137.stdout:3683: writing vpm1375878-144 from 2471613 to 3251896 tid 3 2014-08-08T10:50:53.424 INFO:tasks.rados.rados.0.vpm137.stdout:3683: writing vpm1375878-144 from 3656722 to 3656723 tid 4 2014-08-08T10:50:53.425 INFO:tasks.rados.rados.0.vpm137.stdout:3676: finishing write tid 2 to vpm1375878-320 2014-08-08T10:50:53.425 INFO:tasks.rados.rados.0.vpm137.stdout:3676: finishing write tid 4 to vpm1375878-320 2014-08-08T10:50:53.426 INFO:tasks.rados.rados.0.vpm137.stdout:3678: finishing write tid 1 to vpm1375878-157 2014-08-08T10:50:53.426 INFO:tasks.rados.rados.0.vpm137.stdout:3678: finishing write tid 2 to vpm1375878-157 2014-08-08T10:50:53.426 INFO:tasks.rados.rados.0.vpm137.stdout:3678: finishing write tid 4 to vpm1375878-157 2014-08-08T10:50:53.426 INFO:tasks.rados.rados.0.vpm137.stdout:3680: finishing write tid 1 to vpm1375878-253 2014-08-08T10:50:53.427 INFO:tasks.rados.rados.0.vpm137.stdout:3680: finishing write tid 2 to vpm1375878-253 2014-08-08T10:50:53.427 INFO:tasks.rados.rados.0.vpm137.stdout:3680: finishing write tid 4 to vpm1375878-253 2014-08-08T10:50:53.427 INFO:tasks.rados.rados.0.vpm137.stdout:3681: finishing write tid 1 to vpm1375878-328 2014-08-08T10:50:53.428 INFO:tasks.rados.rados.0.vpm137.stdout:3681: finishing write tid 2 to vpm1375878-328 2014-08-08T10:50:53.428 INFO:tasks.rados.rados.0.vpm137.stdout:3681: finishing write tid 3 to vpm1375878-328 2014-08-08T10:50:53.429 INFO:tasks.rados.rados.0.vpm137.stdout:3676: finishing write tid 5 to vpm1375878-320 2014-08-08T10:50:53.429 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 320 v 422 (ObjNum 2251 snap 0 seq_num 2251) dirty exists 2014-08-08T10:50:53.429 INFO:tasks.rados.rados.0.vpm137.stdout:3681: finishing write tid 5 to vpm1375878-328 2014-08-08T10:50:53.429 INFO:tasks.rados.rados.0.vpm137.stdout:3678: finishing write tid 5 to vpm1375878-157 2014-08-08T10:50:53.429 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 157 v 425 (ObjNum 2252 snap 0 seq_num 2252) dirty exists 2014-08-08T10:50:53.430 INFO:tasks.rados.rados.0.vpm137.stdout:3680: finishing write tid 5 to vpm1375878-253 2014-08-08T10:50:53.430 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 253 v 445 (ObjNum 2253 snap 0 seq_num 2253) dirty exists 2014-08-08T10:50:53.430 INFO:tasks.rados.rados.0.vpm137.stdout:3681: finishing write tid 6 to vpm1375878-328 2014-08-08T10:50:53.430 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 328 v 369 (ObjNum 2254 snap 0 seq_num 2254) dirty exists 2014-08-08T10:50:53.430 INFO:tasks.rados.rados.0.vpm137.stdout:3676: done (4 left) 2014-08-08T10:50:53.432 INFO:tasks.rados.rados.0.vpm137.stdout:3678: done (3 left) 2014-08-08T10:50:53.432 INFO:tasks.rados.rados.0.vpm137.stdout:3680: done (2 left) 2014-08-08T10:50:53.432 INFO:tasks.rados.rados.0.vpm137.stdout:3681: done (1 left) 2014-08-08T10:50:53.433 INFO:tasks.rados.rados.0.vpm137.stdout:3684: write oid 191 current snap is 0 2014-08-08T10:50:53.433 INFO:tasks.rados.rados.0.vpm137.stdout:3684: seq_num 2257 ranges {401551=1} 2014-08-08T10:50:53.433 INFO:tasks.rados.rados.0.vpm137.stdout:3684: writing vpm1375878-191 from 401551 to 401552 tid 1 2014-08-08T10:50:53.434 INFO:tasks.rados.rados.0.vpm137.stdout:3685: read oid 473 snap -1 2014-08-08T10:50:53.434 INFO:tasks.rados.rados.0.vpm137.stdout:3686: write oid 494 current snap is 0 2014-08-08T10:50:53.434 INFO:tasks.rados.rados.0.vpm137.stdout:3686: seq_num 2258 ranges {96781=1} 2014-08-08T10:50:53.436 INFO:tasks.rados.rados.0.vpm137.stdout:3686: writing vpm1375878-494 from 96781 to 96782 tid 1 2014-08-08T10:50:53.437 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 1 to vpm1375878-144 2014-08-08T10:50:53.437 INFO:tasks.rados.rados.0.vpm137.stdout:3687: delete oid 215 current snap is 0 2014-08-08T10:50:53.462 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 2 to vpm1375878-144 2014-08-08T10:50:53.473 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 3 to vpm1375878-144 2014-08-08T10:50:53.513 INFO:tasks.rados.rados.0.vpm137.stdout:3684: finishing write tid 1 to vpm1375878-191 2014-08-08T10:50:53.546 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 4 to vpm1375878-144 2014-08-08T10:50:53.609 INFO:tasks.rados.rados.0.vpm137.stdout:3684: finishing write tid 3 to vpm1375878-191 2014-08-08T10:50:53.610 INFO:tasks.rados.rados.0.vpm137.stdout:3684: finishing write tid 4 to vpm1375878-191 2014-08-08T10:50:53.610 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 191 v 348 (ObjNum 2257 snap 0 seq_num 2257) dirty exists 2014-08-08T10:50:53.755 INFO:tasks.rados.rados.0.vpm137.stdout:3686: finishing write tid 1 to vpm1375878-494 2014-08-08T10:50:53.757 INFO:tasks.rados.rados.0.vpm137.stdout:3686: finishing write tid 3 to vpm1375878-494 2014-08-08T10:50:53.757 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 6 to vpm1375878-144 2014-08-08T10:50:53.758 INFO:tasks.rados.rados.0.vpm137.stdout:3684: done (4 left) 2014-08-08T10:50:53.758 INFO:tasks.rados.rados.0.vpm137.stdout:3687: done (3 left) 2014-08-08T10:50:53.758 INFO:tasks.rados.rados.0.vpm137.stdout:3688: read oid 398 snap -1 2014-08-08T10:50:53.758 INFO:tasks.rados.rados.0.vpm137.stdout:3689: write oid 253 current snap is 0 2014-08-08T10:50:53.759 INFO:tasks.rados.rados.0.vpm137.stdout:3689: seq_num 2260 ranges {501770=667607,1951432=150338} 2014-08-08T10:50:53.804 INFO:tasks.rados.rados.0.vpm137.stdout:3689: writing vpm1375878-253 from 501770 to 1169377 tid 1 2014-08-08T10:50:53.829 INFO:tasks.rados.rados.0.vpm137.stdout:3689: writing vpm1375878-253 from 1951432 to 2101770 tid 2 2014-08-08T10:50:53.830 INFO:tasks.rados.rados.0.vpm137.stdout:3688: expect (ObjNum 1559 snap 0 seq_num 1559) 2014-08-08T10:50:54.486 INFO:tasks.rados.rados.0.vpm137.stdout:3688: done (4 left) 2014-08-08T10:50:54.486 INFO:tasks.rados.rados.0.vpm137.stdout:3690: write oid 14 current snap is 0 2014-08-08T10:50:54.486 INFO:tasks.rados.rados.0.vpm137.stdout:3690: seq_num 2261 ranges {596999=639412,1664540=132459} 2014-08-08T10:50:54.521 INFO:tasks.rados.rados.0.vpm137.stdout:3690: writing vpm1375878-14 from 596999 to 1236411 tid 1 2014-08-08T10:50:54.537 INFO:tasks.rados.rados.0.vpm137.stdout:3690: writing vpm1375878-14 from 1664540 to 1796999 tid 2 2014-08-08T10:50:54.539 INFO:tasks.rados.rados.0.vpm137.stdout:3685: expect (ObjNum 1876 snap 0 seq_num 1876) 2014-08-08T10:50:55.389 INFO:tasks.rados.rados.0.vpm137.stdout:3685: done (4 left) 2014-08-08T10:50:55.389 INFO:tasks.rados.rados.0.vpm137.stdout:3691: read oid 182 snap -1 2014-08-08T10:50:55.389 INFO:tasks.rados.rados.0.vpm137.stdout:3692: read oid 436 snap -1 2014-08-08T10:50:55.390 INFO:tasks.rados.rados.0.vpm137.stdout:3693: write oid 124 current snap is 0 2014-08-08T10:50:55.390 INFO:tasks.rados.rados.0.vpm137.stdout:3693: seq_num 2262 ranges {541829=762640,1778672=542840,2541828=1} 2014-08-08T10:50:55.469 INFO:tasks.rados.rados.0.vpm137.stdout:3693: writing vpm1375878-124 from 541829 to 1304469 tid 1 2014-08-08T10:50:55.510 INFO:tasks.rados.rados.0.vpm137.stdout:3693: writing vpm1375878-124 from 1778672 to 2321512 tid 2 2014-08-08T10:50:55.514 INFO:tasks.rados.rados.0.vpm137.stdout:3693: writing vpm1375878-124 from 2541828 to 2541829 tid 3 2014-08-08T10:50:55.515 INFO:tasks.rados.rados.0.vpm137.stdout:3686: finishing write tid 4 to vpm1375878-494 2014-08-08T10:50:55.515 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 494 v 350 (ObjNum 2258 snap 0 seq_num 2258) dirty exists 2014-08-08T10:50:55.516 INFO:tasks.rados.rados.0.vpm137.stdout:3683: finishing write tid 7 to vpm1375878-144 2014-08-08T10:50:55.516 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 144 v 413 (ObjNum 2256 snap 0 seq_num 2256) dirty exists 2014-08-08T10:50:55.516 INFO:tasks.rados.rados.0.vpm137.stdout:3689: finishing write tid 1 to vpm1375878-253 2014-08-08T10:50:55.516 INFO:tasks.rados.rados.0.vpm137.stdout:3689: finishing write tid 2 to vpm1375878-253 2014-08-08T10:50:55.517 INFO:tasks.rados.rados.0.vpm137.stdout:3689: finishing write tid 4 to vpm1375878-253 2014-08-08T10:50:55.517 INFO:tasks.rados.rados.0.vpm137.stdout:3689: finishing write tid 5 to vpm1375878-253 2014-08-08T10:50:55.517 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 253 v 448 (ObjNum 2260 snap 0 seq_num 2260) dirty exists 2014-08-08T10:50:55.517 INFO:tasks.rados.rados.0.vpm137.stdout:3690: finishing write tid 1 to vpm1375878-14 2014-08-08T10:50:55.518 INFO:tasks.rados.rados.0.vpm137.stdout:3690: finishing write tid 2 to vpm1375878-14 2014-08-08T10:50:55.518 INFO:tasks.rados.rados.0.vpm137.stdout:3690: finishing write tid 4 to vpm1375878-14 2014-08-08T10:50:55.518 INFO:tasks.rados.rados.0.vpm137.stdout:3690: finishing write tid 5 to vpm1375878-14 2014-08-08T10:50:55.518 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 14 v 417 (ObjNum 2261 snap 0 seq_num 2261) dirty exists 2014-08-08T10:50:55.518 INFO:tasks.rados.rados.0.vpm137.stdout:3692: expect (ObjNum 1511 snap 0 seq_num 1511) 2014-08-08T10:50:55.912 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:58.228 INFO:tasks.rados.rados.0.vpm137.stdout:3683: done (6 left) 2014-08-08T10:50:58.229 INFO:tasks.rados.rados.0.vpm137.stdout:3686: done (5 left) 2014-08-08T10:50:58.229 INFO:tasks.rados.rados.0.vpm137.stdout:3689: done (4 left) 2014-08-08T10:50:58.229 INFO:tasks.rados.rados.0.vpm137.stdout:3690: done (3 left) 2014-08-08T10:50:58.229 INFO:tasks.rados.rados.0.vpm137.stdout:3691: done (2 left) 2014-08-08T10:50:58.230 INFO:tasks.rados.rados.0.vpm137.stdout:3692: done (1 left) 2014-08-08T10:50:58.230 INFO:tasks.rados.rados.0.vpm137.stdout:3694: delete oid 164 current snap is 0 2014-08-08T10:50:58.230 INFO:tasks.rados.rados.0.vpm137.stdout:3693: finishing write tid 1 to vpm1375878-124 2014-08-08T10:50:58.231 INFO:tasks.rados.rados.0.vpm137.stdout:3693: finishing write tid 2 to vpm1375878-124 2014-08-08T10:50:58.231 INFO:tasks.rados.rados.0.vpm137.stdout:3693: finishing write tid 3 to vpm1375878-124 2014-08-08T10:50:58.231 INFO:tasks.rados.rados.0.vpm137.stdout:3693: finishing write tid 5 to vpm1375878-124 2014-08-08T10:50:58.232 INFO:tasks.rados.rados.0.vpm137.stdout:3693: finishing write tid 6 to vpm1375878-124 2014-08-08T10:50:58.232 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 124 v 313 (ObjNum 2262 snap 0 seq_num 2262) dirty exists 2014-08-08T10:50:58.245 INFO:tasks.rados.rados.0.vpm137.stdout:3693: done (1 left) 2014-08-08T10:50:58.245 INFO:tasks.rados.rados.0.vpm137.stdout:3694: done (0 left) 2014-08-08T10:50:58.245 INFO:tasks.rados.rados.0.vpm137.stdout:3695: read oid 265 snap -1 2014-08-08T10:50:58.246 INFO:tasks.rados.rados.0.vpm137.stdout:3696: write oid 107 current snap is 0 2014-08-08T10:50:58.246 INFO:tasks.rados.rados.0.vpm137.stdout:3696: seq_num 2264 ranges {698241=742345,1971865=683230,3359367=138874} 2014-08-08T10:50:58.290 INFO:tasks.rados.rados.0.vpm137.stdout:3696: writing vpm1375878-107 from 698241 to 1440586 tid 1 2014-08-08T10:50:58.331 INFO:tasks.rados.rados.0.vpm137.stdout:3696: writing vpm1375878-107 from 1971865 to 2655095 tid 2 2014-08-08T10:50:58.346 INFO:tasks.rados.rados.0.vpm137.stdout:3696: writing vpm1375878-107 from 3359367 to 3498241 tid 3 2014-08-08T10:50:58.348 INFO:tasks.rados.rados.0.vpm137.stdout:3695: expect (ObjNum 1990 snap 0 seq_num 1990) 2014-08-08T10:50:58.939 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:50:58.990 INFO:tasks.rados.rados.0.vpm137.stdout:3695: done (1 left) 2014-08-08T10:50:58.990 INFO:tasks.rados.rados.0.vpm137.stdout:3697: write oid 2 current snap is 0 2014-08-08T10:50:58.991 INFO:tasks.rados.rados.0.vpm137.stdout:3697: seq_num 2265 ranges {243069=1} 2014-08-08T10:50:58.994 INFO:tasks.rados.rados.0.vpm137.stdout:3697: writing vpm1375878-2 from 243069 to 243070 tid 1 2014-08-08T10:50:58.995 INFO:tasks.rados.rados.0.vpm137.stdout:3698: delete oid 303 current snap is 0 2014-08-08T10:50:58.995 INFO:tasks.rados.rados.0.vpm137.stdout:3696: finishing write tid 1 to vpm1375878-107 2014-08-08T10:50:58.996 INFO:tasks.rados.rados.0.vpm137.stdout:3696: finishing write tid 2 to vpm1375878-107 2014-08-08T10:50:58.996 INFO:tasks.rados.rados.0.vpm137.stdout:3696: finishing write tid 3 to vpm1375878-107 2014-08-08T10:50:59.017 INFO:tasks.rados.rados.0.vpm137.stdout:3696: finishing write tid 5 to vpm1375878-107 2014-08-08T10:50:59.049 INFO:tasks.rados.rados.0.vpm137.stdout:3697: finishing write tid 1 to vpm1375878-2 2014-08-08T10:50:59.049 INFO:tasks.rados.rados.0.vpm137.stdout:3697: finishing write tid 3 to vpm1375878-2 2014-08-08T10:50:59.055 INFO:tasks.rados.rados.0.vpm137.stdout:3698: done (2 left) 2014-08-08T10:50:59.055 INFO:tasks.rados.rados.0.vpm137.stdout:3699: write oid 355 current snap is 0 2014-08-08T10:50:59.056 INFO:tasks.rados.rados.0.vpm137.stdout:3699: seq_num 2267 ranges {682105=1} 2014-08-08T10:50:59.063 INFO:tasks.rados.rados.0.vpm137.stdout:3699: writing vpm1375878-355 from 682105 to 682106 tid 1 2014-08-08T10:50:59.064 INFO:tasks.rados.rados.0.vpm137.stdout:3700: write oid 73 current snap is 0 2014-08-08T10:50:59.065 INFO:tasks.rados.rados.0.vpm137.stdout:3700: seq_num 2268 ranges {494711=400000} 2014-08-08T10:50:59.093 INFO:tasks.rados.rados.0.vpm137.stdout:3700: writing vpm1375878-73 from 494711 to 894711 tid 1 2014-08-08T10:50:59.095 INFO:tasks.rados.rados.0.vpm137.stdout:3697: finishing write tid 4 to vpm1375878-2 2014-08-08T10:50:59.095 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 2 v 409 (ObjNum 2265 snap 0 seq_num 2265) dirty exists 2014-08-08T10:50:59.095 INFO:tasks.rados.rados.0.vpm137.stdout:3697: done (3 left) 2014-08-08T10:50:59.095 INFO:tasks.rados.rados.0.vpm137.stdout:3701: write oid 9 current snap is 0 2014-08-08T10:50:59.096 INFO:tasks.rados.rados.0.vpm137.stdout:3701: seq_num 2269 ranges {687093=637462,2015037=420702,2687092=1} 2014-08-08T10:50:59.143 INFO:tasks.rados.rados.0.vpm137.stdout:3701: writing vpm1375878-9 from 687093 to 1324555 tid 1 2014-08-08T10:50:59.178 INFO:tasks.rados.rados.0.vpm137.stdout:3701: writing vpm1375878-9 from 2015037 to 2435739 tid 2 2014-08-08T10:50:59.184 INFO:tasks.rados.rados.0.vpm137.stdout:3701: writing vpm1375878-9 from 2687092 to 2687093 tid 3 2014-08-08T10:50:59.185 INFO:tasks.rados.rados.0.vpm137.stdout:3696: finishing write tid 6 to vpm1375878-107 2014-08-08T10:50:59.185 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 107 v 452 (ObjNum 2264 snap 0 seq_num 2264) dirty exists 2014-08-08T10:50:59.185 INFO:tasks.rados.rados.0.vpm137.stdout:3699: finishing write tid 1 to vpm1375878-355 2014-08-08T10:50:59.186 INFO:tasks.rados.rados.0.vpm137.stdout:3699: finishing write tid 3 to vpm1375878-355 2014-08-08T10:50:59.186 INFO:tasks.rados.rados.0.vpm137.stdout:3699: finishing write tid 4 to vpm1375878-355 2014-08-08T10:50:59.186 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 355 v 352 (ObjNum 2267 snap 0 seq_num 2267) dirty exists 2014-08-08T10:50:59.186 INFO:tasks.rados.rados.0.vpm137.stdout:3696: done (3 left) 2014-08-08T10:50:59.186 INFO:tasks.rados.rados.0.vpm137.stdout:3699: done (2 left) 2014-08-08T10:50:59.187 INFO:tasks.rados.rados.0.vpm137.stdout:3702: delete oid 353 current snap is 0 2014-08-08T10:50:59.260 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:50:59.305 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:50:59.373 INFO:tasks.rados.rados.0.vpm137.stdout:3700: finishing write tid 1 to vpm1375878-73 2014-08-08T10:50:59.392 INFO:tasks.rados.rados.0.vpm137.stdout:3700: finishing write tid 3 to vpm1375878-73 2014-08-08T10:50:59.396 INFO:tasks.rados.rados.0.vpm137.stdout:3700: finishing write tid 4 to vpm1375878-73 2014-08-08T10:50:59.396 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 73 v 409 (ObjNum 2268 snap 0 seq_num 2268) dirty exists 2014-08-08T10:50:59.400 INFO:tasks.rados.rados.0.vpm137.stdout:3701: finishing write tid 1 to vpm1375878-9 2014-08-08T10:50:59.401 INFO:tasks.rados.rados.0.vpm137.stdout:3701: finishing write tid 2 to vpm1375878-9 2014-08-08T10:50:59.402 INFO:tasks.rados.rados.0.vpm137.stdout:3701: finishing write tid 3 to vpm1375878-9 2014-08-08T10:50:59.402 INFO:tasks.rados.rados.0.vpm137.stdout:3701: finishing write tid 5 to vpm1375878-9 2014-08-08T10:50:59.403 INFO:tasks.rados.rados.0.vpm137.stdout:3700: done (2 left) 2014-08-08T10:50:59.404 INFO:tasks.rados.rados.0.vpm137.stdout:3702: done (1 left) 2014-08-08T10:50:59.404 INFO:tasks.rados.rados.0.vpm137.stdout:3703: write oid 235 current snap is 0 2014-08-08T10:50:59.405 INFO:tasks.rados.rados.0.vpm137.stdout:3703: seq_num 2271 ranges {478577=733519,1711078=367499} 2014-08-08T10:50:59.437 INFO:tasks.rados.rados.0.vpm137.stdout:3703: writing vpm1375878-235 from 478577 to 1212096 tid 1 2014-08-08T10:50:59.461 INFO:tasks.rados.rados.0.vpm137.stdout:3703: writing vpm1375878-235 from 1711078 to 2078577 tid 2 2014-08-08T10:50:59.462 INFO:tasks.rados.rados.0.vpm137.stdout:3704: write oid 24 current snap is 0 2014-08-08T10:50:59.462 INFO:tasks.rados.rados.0.vpm137.stdout:3704: seq_num 2272 ranges {671982=456747,1922361=782712,3295698=525028,3871981=1} 2014-08-08T10:50:59.490 INFO:tasks.rados.rados.0.vpm137.stdout:3704: writing vpm1375878-24 from 671982 to 1128729 tid 1 2014-08-08T10:50:59.542 INFO:tasks.rados.rados.0.vpm137.stdout:3704: writing vpm1375878-24 from 1922361 to 2705073 tid 2 2014-08-08T10:50:59.581 INFO:tasks.rados.rados.0.vpm137.stdout:3704: writing vpm1375878-24 from 3295698 to 3820726 tid 3 2014-08-08T10:50:59.586 INFO:tasks.rados.rados.0.vpm137.stdout:3704: writing vpm1375878-24 from 3871981 to 3871982 tid 4 2014-08-08T10:50:59.588 INFO:tasks.rados.rados.0.vpm137.stdout:3703: finishing write tid 1 to vpm1375878-235 2014-08-08T10:50:59.589 INFO:tasks.rados.rados.0.vpm137.stdout:3701: finishing write tid 6 to vpm1375878-9 2014-08-08T10:50:59.589 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 9 v 498 (ObjNum 2269 snap 0 seq_num 2269) dirty exists 2014-08-08T10:50:59.590 INFO:tasks.rados.rados.0.vpm137.stdout:3703: finishing write tid 2 to vpm1375878-235 2014-08-08T10:50:59.590 INFO:tasks.rados.rados.0.vpm137.stdout:3703: finishing write tid 4 to vpm1375878-235 2014-08-08T10:50:59.590 INFO:tasks.rados.rados.0.vpm137.stdout:3703: finishing write tid 5 to vpm1375878-235 2014-08-08T10:50:59.590 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 235 v 428 (ObjNum 2271 snap 0 seq_num 2271) dirty exists 2014-08-08T10:50:59.591 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 1 to vpm1375878-24 2014-08-08T10:50:59.591 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 2 to vpm1375878-24 2014-08-08T10:50:59.591 INFO:tasks.rados.rados.0.vpm137.stdout:3701: done (2 left) 2014-08-08T10:50:59.592 INFO:tasks.rados.rados.0.vpm137.stdout:3703: done (1 left) 2014-08-08T10:50:59.592 INFO:tasks.rados.rados.0.vpm137.stdout:3705: write oid 400 current snap is 0 2014-08-08T10:50:59.592 INFO:tasks.rados.rados.0.vpm137.stdout:3705: seq_num 2273 ranges {84587=1} 2014-08-08T10:50:59.593 INFO:tasks.rados.rados.0.vpm137.stdout:3705: writing vpm1375878-400 from 84587 to 84588 tid 1 2014-08-08T10:50:59.593 INFO:tasks.rados.rados.0.vpm137.stdout:3706: write oid 159 current snap is 0 2014-08-08T10:50:59.593 INFO:tasks.rados.rados.0.vpm137.stdout:3706: seq_num 2274 ranges {579818=436452,1617378=522078,2917349=545137,3779817=1} 2014-08-08T10:50:59.622 INFO:tasks.rados.rados.0.vpm137.stdout:3706: writing vpm1375878-159 from 579818 to 1016270 tid 1 2014-08-08T10:50:59.647 INFO:tasks.rados.rados.0.vpm137.stdout:3706: writing vpm1375878-159 from 1617378 to 2139456 tid 2 2014-08-08T10:50:59.674 INFO:tasks.rados.rados.0.vpm137.stdout:3706: writing vpm1375878-159 from 2917349 to 3462486 tid 3 2014-08-08T10:50:59.678 INFO:tasks.rados.rados.0.vpm137.stdout:3706: writing vpm1375878-159 from 3779817 to 3779818 tid 4 2014-08-08T10:50:59.678 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 3 to vpm1375878-24 2014-08-08T10:50:59.679 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 4 to vpm1375878-24 2014-08-08T10:50:59.679 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 6 to vpm1375878-24 2014-08-08T10:50:59.679 INFO:tasks.rados.rados.0.vpm137.stdout:3704: finishing write tid 7 to vpm1375878-24 2014-08-08T10:50:59.680 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 24 v 457 (ObjNum 2272 snap 0 seq_num 2272) dirty exists 2014-08-08T10:50:59.680 INFO:tasks.rados.rados.0.vpm137.stdout:3705: finishing write tid 1 to vpm1375878-400 2014-08-08T10:50:59.680 INFO:tasks.rados.rados.0.vpm137.stdout:3705: finishing write tid 3 to vpm1375878-400 2014-08-08T10:50:59.680 INFO:tasks.rados.rados.0.vpm137.stdout:3705: finishing write tid 4 to vpm1375878-400 2014-08-08T10:50:59.681 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 400 v 500 (ObjNum 2273 snap 0 seq_num 2273) dirty exists 2014-08-08T10:50:59.681 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 1 to vpm1375878-159 2014-08-08T10:50:59.682 INFO:tasks.rados.rados.0.vpm137.stdout:3704: done (2 left) 2014-08-08T10:50:59.682 INFO:tasks.rados.rados.0.vpm137.stdout:3705: done (1 left) 2014-08-08T10:50:59.683 INFO:tasks.rados.rados.0.vpm137.stdout:3707: read oid 32 snap -1 2014-08-08T10:50:59.683 INFO:tasks.rados.rados.0.vpm137.stdout:3708: read oid 153 snap -1 2014-08-08T10:50:59.683 INFO:tasks.rados.rados.0.vpm137.stdout:3709: write oid 400 current snap is 0 2014-08-08T10:50:59.683 INFO:tasks.rados.rados.0.vpm137.stdout:3709: seq_num 2275 ranges {772200=656832,1572199=1} 2014-08-08T10:50:59.714 INFO:tasks.rados.rados.0.vpm137.stdout:3709: writing vpm1375878-400 from 772200 to 1429032 tid 1 2014-08-08T10:50:59.718 INFO:tasks.rados.rados.0.vpm137.stdout:3709: writing vpm1375878-400 from 1572199 to 1572200 tid 2 2014-08-08T10:50:59.719 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 2 to vpm1375878-159 2014-08-08T10:50:59.719 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 3 to vpm1375878-159 2014-08-08T10:50:59.719 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 4 to vpm1375878-159 2014-08-08T10:50:59.719 INFO:tasks.rados.rados.0.vpm137.stdout:3710: read oid 52 snap -1 2014-08-08T10:50:59.720 INFO:tasks.rados.rados.0.vpm137.stdout:3711: read oid 294 snap -1 2014-08-08T10:50:59.720 INFO:tasks.rados.rados.0.vpm137.stdout:3712: read oid 162 snap -1 2014-08-08T10:50:59.721 INFO:tasks.rados.rados.0.vpm137.stdout:3713: write oid 450 current snap is 0 2014-08-08T10:50:59.721 INFO:tasks.rados.rados.0.vpm137.stdout:3713: seq_num 2276 ranges {467429=780061,1267428=1} 2014-08-08T10:50:59.771 INFO:tasks.rados.rados.0.vpm137.stdout:3713: writing vpm1375878-450 from 467429 to 1247490 tid 1 2014-08-08T10:50:59.776 INFO:tasks.rados.rados.0.vpm137.stdout:3713: writing vpm1375878-450 from 1267428 to 1267429 tid 2 2014-08-08T10:50:59.777 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 6 to vpm1375878-159 2014-08-08T10:50:59.777 INFO:tasks.rados.rados.0.vpm137.stdout:3706: finishing write tid 7 to vpm1375878-159 2014-08-08T10:50:59.777 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 159 v 462 (ObjNum 2274 snap 0 seq_num 2274) dirty exists 2014-08-08T10:50:59.777 INFO:tasks.rados.rados.0.vpm137.stdout:3709: finishing write tid 1 to vpm1375878-400 2014-08-08T10:50:59.778 INFO:tasks.rados.rados.0.vpm137.stdout:3709: finishing write tid 2 to vpm1375878-400 2014-08-08T10:50:59.778 INFO:tasks.rados.rados.0.vpm137.stdout:3706: done (7 left) 2014-08-08T10:50:59.778 INFO:tasks.rados.rados.0.vpm137.stdout:3708: done (6 left) 2014-08-08T10:50:59.778 INFO:tasks.rados.rados.0.vpm137.stdout:3714: write oid 27 current snap is 0 2014-08-08T10:50:59.778 INFO:tasks.rados.rados.0.vpm137.stdout:3714: seq_num 2277 ranges {680035=636537,1480034=1} 2014-08-08T10:50:59.821 INFO:tasks.rados.rados.0.vpm137.stdout:3714: writing vpm1375878-27 from 680035 to 1316572 tid 1 2014-08-08T10:50:59.826 INFO:tasks.rados.rados.0.vpm137.stdout:3714: writing vpm1375878-27 from 1480034 to 1480035 tid 2 2014-08-08T10:50:59.827 INFO:tasks.rados.rados.0.vpm137.stdout:3707: expect (ObjNum 1589 snap 0 seq_num 1589) 2014-08-08T10:51:00.306 INFO:tasks.rados.rados.0.vpm137.stdout:3707: done (6 left) 2014-08-08T10:51:00.306 INFO:tasks.rados.rados.0.vpm137.stdout:3715: read oid 247 snap -1 2014-08-08T10:51:00.306 INFO:tasks.rados.rados.0.vpm137.stdout:3716: read oid 139 snap -1 2014-08-08T10:51:00.307 INFO:tasks.rados.rados.0.vpm137.stdout:3717: write oid 29 current snap is 0 2014-08-08T10:51:00.307 INFO:tasks.rados.rados.0.vpm137.stdout:3717: seq_num 2278 ranges {473441=759765,1948466=589355,3000202=273239} 2014-08-08T10:51:00.349 INFO:tasks.rados.rados.0.vpm137.stdout:3717: writing vpm1375878-29 from 473441 to 1233206 tid 1 2014-08-08T10:51:00.376 INFO:tasks.rados.rados.0.vpm137.stdout:3717: writing vpm1375878-29 from 1948466 to 2537821 tid 2 2014-08-08T10:51:00.390 INFO:tasks.rados.rados.0.vpm137.stdout:3717: writing vpm1375878-29 from 3000202 to 3273441 tid 3 2014-08-08T10:51:00.391 INFO:tasks.rados.rados.0.vpm137.stdout:3709: finishing write tid 4 to vpm1375878-400 2014-08-08T10:51:00.392 INFO:tasks.rados.rados.0.vpm137.stdout:3709: finishing write tid 5 to vpm1375878-400 2014-08-08T10:51:00.392 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 400 v 503 (ObjNum 2275 snap 0 seq_num 2275) dirty exists 2014-08-08T10:51:00.392 INFO:tasks.rados.rados.0.vpm137.stdout:3713: finishing write tid 1 to vpm1375878-450 2014-08-08T10:51:00.393 INFO:tasks.rados.rados.0.vpm137.stdout:3710: expect (ObjNum 2207 snap 0 seq_num 2207) 2014-08-08T10:51:00.424 INFO:tasks.rados.rados.0.vpm137.stdout:3709: done (8 left) 2014-08-08T10:51:00.424 INFO:tasks.rados.rados.0.vpm137.stdout:3710: done (7 left) 2014-08-08T10:51:00.424 INFO:tasks.rados.rados.0.vpm137.stdout:3718: write oid 416 current snap is 0 2014-08-08T10:51:00.424 INFO:tasks.rados.rados.0.vpm137.stdout:3718: seq_num 2279 ranges {568671=731570,2061575=659550,2968670=1} 2014-08-08T10:51:00.451 INFO:tasks.rados.rados.0.vpm137.stdout:3718: writing vpm1375878-416 from 568671 to 1300241 tid 1 2014-08-08T10:51:00.508 INFO:tasks.rados.rados.0.vpm137.stdout:3718: writing vpm1375878-416 from 2061575 to 2721125 tid 2 2014-08-08T10:51:00.512 INFO:tasks.rados.rados.0.vpm137.stdout:3718: writing vpm1375878-416 from 2968670 to 2968671 tid 3 2014-08-08T10:51:00.516 INFO:tasks.rados.rados.0.vpm137.stdout:3713: finishing write tid 2 to vpm1375878-450 2014-08-08T10:51:00.516 INFO:tasks.rados.rados.0.vpm137.stdout:3713: finishing write tid 4 to vpm1375878-450 2014-08-08T10:51:00.516 INFO:tasks.rados.rados.0.vpm137.stdout:3714: finishing write tid 1 to vpm1375878-27 2014-08-08T10:51:00.516 INFO:tasks.rados.rados.0.vpm137.stdout:3714: finishing write tid 2 to vpm1375878-27 2014-08-08T10:51:00.517 INFO:tasks.rados.rados.0.vpm137.stdout:3714: finishing write tid 4 to vpm1375878-27 2014-08-08T10:51:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:3713: finishing write tid 5 to vpm1375878-450 2014-08-08T10:51:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 450 v 310 (ObjNum 2276 snap 0 seq_num 2276) dirty exists 2014-08-08T10:51:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:3714: finishing write tid 5 to vpm1375878-27 2014-08-08T10:51:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 27 v 604 (ObjNum 2277 snap 0 seq_num 2277) dirty exists 2014-08-08T10:51:00.522 INFO:tasks.rados.rados.0.vpm137.stdout:3712: expect (ObjNum 972 snap 0 seq_num 972) 2014-08-08T10:51:02.501 INFO:tasks.rados.rados.0.vpm137.stdout:3712: done (7 left) 2014-08-08T10:51:02.501 INFO:tasks.rados.rados.0.vpm137.stdout:3713: done (6 left) 2014-08-08T10:51:02.502 INFO:tasks.rados.rados.0.vpm137.stdout:3714: done (5 left) 2014-08-08T10:51:02.502 INFO:tasks.rados.rados.0.vpm137.stdout:3715: done (4 left) 2014-08-08T10:51:02.502 INFO:tasks.rados.rados.0.vpm137.stdout:3719: read oid 48 snap -1 2014-08-08T10:51:02.502 INFO:tasks.rados.rados.0.vpm137.stdout:3720: write oid 70 current snap is 0 2014-08-08T10:51:02.503 INFO:tasks.rados.rados.0.vpm137.stdout:3720: seq_num 2280 ranges {664924=454798,1643482=729745,2664923=1} 2014-08-08T10:51:02.531 INFO:tasks.rados.rados.0.vpm137.stdout:3720: writing vpm1375878-70 from 664924 to 1119722 tid 1 2014-08-08T10:51:02.580 INFO:tasks.rados.rados.0.vpm137.stdout:3720: writing vpm1375878-70 from 1643482 to 2373227 tid 2 2014-08-08T10:51:02.584 INFO:tasks.rados.rados.0.vpm137.stdout:3720: writing vpm1375878-70 from 2664923 to 2664924 tid 3 2014-08-08T10:51:02.585 INFO:tasks.rados.rados.0.vpm137.stdout:3718: finishing write tid 1 to vpm1375878-416 2014-08-08T10:51:02.585 INFO:tasks.rados.rados.0.vpm137.stdout:3711: expect (ObjNum 995 snap 0 seq_num 995) 2014-08-08T10:51:02.701 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 842374579, u'bytes_used': 1752956928, u'op_per_sec': 19, u'version': 1682, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1527484, u'write_bytes_sec': 2243447, u'bytes_avail': 427534057472}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:02.701 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:02.903 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:02.903 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:03.092 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:04.561 INFO:tasks.rados.rados.0.vpm137.stdout:3711: done (5 left) 2014-08-08T10:51:04.561 INFO:tasks.rados.rados.0.vpm137.stdout:3721: read oid 337 snap -1 2014-08-08T10:51:04.561 INFO:tasks.rados.rados.0.vpm137.stdout:3722: write oid 193 current snap is 0 2014-08-08T10:51:04.562 INFO:tasks.rados.rados.0.vpm137.stdout:3722: seq_num 2281 ranges {608730=711274,1737390=551365,2782630=626100} 2014-08-08T10:51:04.590 INFO:tasks.rados.rados.0.vpm137.stdout:3722: writing vpm1375878-193 from 608730 to 1320004 tid 1 2014-08-08T10:51:04.621 INFO:tasks.rados.rados.0.vpm137.stdout:3722: writing vpm1375878-193 from 1737390 to 2288755 tid 2 2014-08-08T10:51:04.659 INFO:tasks.rados.rados.0.vpm137.stdout:3722: writing vpm1375878-193 from 2782630 to 3408730 tid 3 2014-08-08T10:51:04.662 INFO:tasks.rados.rados.0.vpm137.stdout:3718: finishing write tid 2 to vpm1375878-416 2014-08-08T10:51:04.663 INFO:tasks.rados.rados.0.vpm137.stdout:3718: finishing write tid 3 to vpm1375878-416 2014-08-08T10:51:04.663 INFO:tasks.rados.rados.0.vpm137.stdout:3718: finishing write tid 5 to vpm1375878-416 2014-08-08T10:51:04.663 INFO:tasks.rados.rados.0.vpm137.stdout:3718: finishing write tid 6 to vpm1375878-416 2014-08-08T10:51:04.664 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 416 v 314 (ObjNum 2279 snap 0 seq_num 2279) dirty exists 2014-08-08T10:51:04.664 INFO:tasks.rados.rados.0.vpm137.stdout:3716: expect (ObjNum 138 snap 0 seq_num 138) 2014-08-08T10:51:06.116 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:06.331 INFO:tasks.rados.rados.0.vpm137.stdout:3716: done (6 left) 2014-08-08T10:51:06.332 INFO:tasks.rados.rados.0.vpm137.stdout:3718: done (5 left) 2014-08-08T10:51:06.332 INFO:tasks.rados.rados.0.vpm137.stdout:3723: delete oid 262 current snap is 0 2014-08-08T10:51:06.332 INFO:tasks.rados.rados.0.vpm137.stdout:3717: finishing write tid 1 to vpm1375878-29 2014-08-08T10:51:06.332 INFO:tasks.rados.rados.0.vpm137.stdout:3717: finishing write tid 2 to vpm1375878-29 2014-08-08T10:51:06.333 INFO:tasks.rados.rados.0.vpm137.stdout:3717: finishing write tid 3 to vpm1375878-29 2014-08-08T10:51:06.333 INFO:tasks.rados.rados.0.vpm137.stdout:3717: finishing write tid 5 to vpm1375878-29 2014-08-08T10:51:06.334 INFO:tasks.rados.rados.0.vpm137.stdout:3717: finishing write tid 6 to vpm1375878-29 2014-08-08T10:51:06.334 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 29 v 413 (ObjNum 2278 snap 0 seq_num 2278) dirty exists 2014-08-08T10:51:06.334 INFO:tasks.rados.rados.0.vpm137.stdout:3719: expect (ObjNum 2107 snap 0 seq_num 2107) 2014-08-08T10:51:06.338 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:06.390 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:06.648 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 852486871, u'bytes_used': 1761398784, u'op_per_sec': 15, u'version': 1683, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1681129, u'write_bytes_sec': 1787020, u'bytes_avail': 427525615616}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:06.649 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:06.871 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:06.871 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:07.076 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:07.371 INFO:tasks.rados.rados.0.vpm137.stdout:3717: done (5 left) 2014-08-08T10:51:07.371 INFO:tasks.rados.rados.0.vpm137.stdout:3719: done (4 left) 2014-08-08T10:51:07.372 INFO:tasks.rados.rados.0.vpm137.stdout:3723: done (3 left) 2014-08-08T10:51:07.373 INFO:tasks.rados.rados.0.vpm137.stdout:3724: delete oid 149 current snap is 0 2014-08-08T10:51:07.373 INFO:tasks.rados.rados.0.vpm137.stdout:3720: finishing write tid 1 to vpm1375878-70 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:3720: finishing write tid 2 to vpm1375878-70 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:3720: finishing write tid 3 to vpm1375878-70 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:3720: finishing write tid 5 to vpm1375878-70 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:3720: finishing write tid 6 to vpm1375878-70 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 70 v 436 (ObjNum 2280 snap 0 seq_num 2280) dirty exists 2014-08-08T10:51:07.381 INFO:tasks.rados.rados.0.vpm137.stdout:3721: expect (ObjNum 1733 snap 0 seq_num 1733) 2014-08-08T10:51:07.417 INFO:tasks.rados.rados.0.vpm137.stdout:3720: done (3 left) 2014-08-08T10:51:07.417 INFO:tasks.rados.rados.0.vpm137.stdout:3721: done (2 left) 2014-08-08T10:51:07.417 INFO:tasks.rados.rados.0.vpm137.stdout:3724: done (1 left) 2014-08-08T10:51:07.417 INFO:tasks.rados.rados.0.vpm137.stdout:3725: read oid 149 snap -1 2014-08-08T10:51:07.418 INFO:tasks.rados.rados.0.vpm137.stdout:3726: read oid 425 snap -1 2014-08-08T10:51:07.418 INFO:tasks.rados.rados.0.vpm137.stdout:3727: write oid 164 current snap is 0 2014-08-08T10:51:07.418 INFO:tasks.rados.rados.0.vpm137.stdout:3727: seq_num 2284 ranges {708947=400000} 2014-08-08T10:51:07.446 INFO:tasks.rados.rados.0.vpm137.stdout:3727: writing vpm1375878-164 from 708947 to 1108947 tid 1 2014-08-08T10:51:07.448 INFO:tasks.rados.rados.0.vpm137.stdout:3722: finishing write tid 1 to vpm1375878-193 2014-08-08T10:51:07.448 INFO:tasks.rados.rados.0.vpm137.stdout:3722: finishing write tid 2 to vpm1375878-193 2014-08-08T10:51:07.449 INFO:tasks.rados.rados.0.vpm137.stdout:3722: finishing write tid 3 to vpm1375878-193 2014-08-08T10:51:07.456 INFO:tasks.rados.rados.0.vpm137.stdout:3722: finishing write tid 5 to vpm1375878-193 2014-08-08T10:51:07.456 INFO:tasks.rados.rados.0.vpm137.stdout:3722: finishing write tid 6 to vpm1375878-193 2014-08-08T10:51:07.456 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 193 v 351 (ObjNum 2281 snap 0 seq_num 2281) dirty exists 2014-08-08T10:51:07.457 INFO:tasks.rados.rados.0.vpm137.stdout:3726: expect (ObjNum 2160 snap 0 seq_num 2160) 2014-08-08T10:51:07.638 INFO:tasks.rados.rados.0.vpm137.stdout:3722: done (3 left) 2014-08-08T10:51:07.638 INFO:tasks.rados.rados.0.vpm137.stdout:3725: done (2 left) 2014-08-08T10:51:07.638 INFO:tasks.rados.rados.0.vpm137.stdout:3726: done (1 left) 2014-08-08T10:51:07.638 INFO:tasks.rados.rados.0.vpm137.stdout:3728: read oid 402 snap -1 2014-08-08T10:51:07.639 INFO:tasks.rados.rados.0.vpm137.stdout:3729: delete oid 247 current snap is 0 2014-08-08T10:51:07.639 INFO:tasks.rados.rados.0.vpm137.stdout:3727: finishing write tid 1 to vpm1375878-164 2014-08-08T10:51:07.639 INFO:tasks.rados.rados.0.vpm137.stdout:3727: finishing write tid 3 to vpm1375878-164 2014-08-08T10:51:07.639 INFO:tasks.rados.rados.0.vpm137.stdout:3727: finishing write tid 4 to vpm1375878-164 2014-08-08T10:51:07.639 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 164 v 606 (ObjNum 2284 snap 0 seq_num 2284) dirty exists 2014-08-08T10:51:07.642 INFO:tasks.rados.rados.0.vpm137.stdout:3727: done (2 left) 2014-08-08T10:51:07.642 INFO:tasks.rados.rados.0.vpm137.stdout:3729: done (1 left) 2014-08-08T10:51:07.643 INFO:tasks.rados.rados.0.vpm137.stdout:3730: write oid 19 current snap is 0 2014-08-08T10:51:07.643 INFO:tasks.rados.rados.0.vpm137.stdout:3730: seq_num 2286 ranges {713935=758840,2201812=405189,3072086=41849} 2014-08-08T10:51:07.673 INFO:tasks.rados.rados.0.vpm137.stdout:3730: writing vpm1375878-19 from 713935 to 1472775 tid 1 2014-08-08T10:51:07.710 INFO:tasks.rados.rados.0.vpm137.stdout:3730: writing vpm1375878-19 from 2201812 to 2607001 tid 2 2014-08-08T10:51:07.716 INFO:tasks.rados.rados.0.vpm137.stdout:3730: writing vpm1375878-19 from 3072086 to 3113935 tid 3 2014-08-08T10:51:07.716 INFO:tasks.rados.rados.0.vpm137.stdout:3728: expect (ObjNum 1461 snap 0 seq_num 1461) 2014-08-08T10:51:08.568 INFO:tasks.rados.rados.0.vpm137.stdout:3728: done (1 left) 2014-08-08T10:51:08.569 INFO:tasks.rados.rados.0.vpm137.stdout:3731: read oid 250 snap -1 2014-08-08T10:51:08.569 INFO:tasks.rados.rados.0.vpm137.stdout:3732: delete oid 281 current snap is 0 2014-08-08T10:51:08.570 INFO:tasks.rados.rados.0.vpm137.stdout:3730: finishing write tid 1 to vpm1375878-19 2014-08-08T10:51:08.570 INFO:tasks.rados.rados.0.vpm137.stdout:3730: finishing write tid 2 to vpm1375878-19 2014-08-08T10:51:08.571 INFO:tasks.rados.rados.0.vpm137.stdout:3730: finishing write tid 3 to vpm1375878-19 2014-08-08T10:51:08.571 INFO:tasks.rados.rados.0.vpm137.stdout:3730: finishing write tid 5 to vpm1375878-19 2014-08-08T10:51:08.572 INFO:tasks.rados.rados.0.vpm137.stdout:3730: finishing write tid 6 to vpm1375878-19 2014-08-08T10:51:08.580 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 19 v 507 (ObjNum 2286 snap 0 seq_num 2286) dirty exists 2014-08-08T10:51:08.581 INFO:tasks.rados.rados.0.vpm137.stdout:3731: expect (ObjNum 1783 snap 0 seq_num 1783) 2014-08-08T10:51:08.615 INFO:tasks.rados.rados.0.vpm137.stdout:3730: done (2 left) 2014-08-08T10:51:08.615 INFO:tasks.rados.rados.0.vpm137.stdout:3731: done (1 left) 2014-08-08T10:51:08.616 INFO:tasks.rados.rados.0.vpm137.stdout:3732: done (0 left) 2014-08-08T10:51:08.616 INFO:tasks.rados.rados.0.vpm137.stdout:3733: read oid 226 snap -1 2014-08-08T10:51:08.616 INFO:tasks.rados.rados.0.vpm137.stdout:3734: read oid 4 snap -1 2014-08-08T10:51:08.616 INFO:tasks.rados.rados.0.vpm137.stdout:3735: read oid 356 snap -1 2014-08-08T10:51:08.617 INFO:tasks.rados.rados.0.vpm137.stdout:3736: write oid 219 current snap is 0 2014-08-08T10:51:08.617 INFO:tasks.rados.rados.0.vpm137.stdout:3736: seq_num 2288 ranges {755018=454897,1996028=429227,3056361=421196,3555017=1} 2014-08-08T10:51:08.649 INFO:tasks.rados.rados.0.vpm137.stdout:3736: writing vpm1375878-219 from 755018 to 1209915 tid 1 2014-08-08T10:51:08.671 INFO:tasks.rados.rados.0.vpm137.stdout:3736: writing vpm1375878-219 from 1996028 to 2425255 tid 2 2014-08-08T10:51:08.689 INFO:tasks.rados.rados.0.vpm137.stdout:3736: writing vpm1375878-219 from 3056361 to 3477557 tid 3 2014-08-08T10:51:08.690 INFO:tasks.rados.rados.0.vpm137.stdout:3736: writing vpm1375878-219 from 3555017 to 3555018 tid 4 2014-08-08T10:51:08.691 INFO:tasks.rados.rados.0.vpm137.stdout:3735: expect (ObjNum 1554 snap 0 seq_num 1554) 2014-08-08T10:51:08.828 INFO:tasks.rados.rados.0.vpm137.stdout:3735: done (3 left) 2014-08-08T10:51:08.829 INFO:tasks.rados.rados.0.vpm137.stdout:3737: read oid 361 snap -1 2014-08-08T10:51:08.829 INFO:tasks.rados.rados.0.vpm137.stdout:3738: delete oid 350 current snap is 0 2014-08-08T10:51:08.829 INFO:tasks.rados.rados.0.vpm137.stdout:3733: expect (ObjNum 1631 snap 0 seq_num 1631) 2014-08-08T10:51:09.301 INFO:tasks.rados.rados.0.vpm137.stdout:3733: done (4 left) 2014-08-08T10:51:09.302 INFO:tasks.rados.rados.0.vpm137.stdout:3738: done (3 left) 2014-08-08T10:51:09.302 INFO:tasks.rados.rados.0.vpm137.stdout:3739: write oid 236 current snap is 0 2014-08-08T10:51:09.302 INFO:tasks.rados.rados.0.vpm137.stdout:3739: seq_num 2290 ranges {795078=433578,1919397=721041,3191036=689881,3995077=1} 2014-08-08T10:51:09.324 INFO:tasks.rados.rados.0.vpm137.stdout:3739: writing vpm1375878-236 from 795078 to 1228656 tid 1 2014-08-08T10:51:09.361 INFO:tasks.rados.rados.0.vpm137.stdout:3739: writing vpm1375878-236 from 1919397 to 2640438 tid 2 2014-08-08T10:51:09.390 INFO:tasks.rados.rados.0.vpm137.stdout:3739: writing vpm1375878-236 from 3191036 to 3880917 tid 3 2014-08-08T10:51:09.394 INFO:tasks.rados.rados.0.vpm137.stdout:3739: writing vpm1375878-236 from 3995077 to 3995078 tid 4 2014-08-08T10:51:09.395 INFO:tasks.rados.rados.0.vpm137.stdout:3734: expect (ObjNum 1935 snap 0 seq_num 1935) 2014-08-08T10:51:10.107 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:10.337 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:10.451 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:10.462 INFO:tasks.rados.rados.0.vpm137.stdout:3734: done (3 left) 2014-08-08T10:51:10.462 INFO:tasks.rados.rados.0.vpm137.stdout:3740: write oid 346 current snap is 0 2014-08-08T10:51:10.463 INFO:tasks.rados.rados.0.vpm137.stdout:3740: seq_num 2291 ranges {454211=406406,1254210=1} 2014-08-08T10:51:10.487 INFO:tasks.rados.rados.0.vpm137.stdout:3740: writing vpm1375878-346 from 454211 to 860617 tid 1 2014-08-08T10:51:10.493 INFO:tasks.rados.rados.0.vpm137.stdout:3740: writing vpm1375878-346 from 1254210 to 1254211 tid 2 2014-08-08T10:51:10.496 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 1 to vpm1375878-219 2014-08-08T10:51:10.496 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 2 to vpm1375878-219 2014-08-08T10:51:10.496 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 3 to vpm1375878-219 2014-08-08T10:51:10.497 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 4 to vpm1375878-219 2014-08-08T10:51:10.497 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 6 to vpm1375878-219 2014-08-08T10:51:10.498 INFO:tasks.rados.rados.0.vpm137.stdout:3736: finishing write tid 7 to vpm1375878-219 2014-08-08T10:51:10.498 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 219 v 418 (ObjNum 2288 snap 0 seq_num 2288) dirty exists 2014-08-08T10:51:10.499 INFO:tasks.rados.rados.0.vpm137.stdout:3737: expect (ObjNum 830 snap 0 seq_num 830) 2014-08-08T10:51:10.812 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:50:13.453597', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89739332, u'health': u'HEALTH_OK', u'kb_used': 9214236, u'store_stats': {u'bytes_total': 25170102, u'bytes_log': 5748242, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:50:17.562071', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89738228, u'health': u'HEALTH_OK', u'kb_used': 9215340, u'store_stats': {u'bytes_total': 28038384, u'bytes_log': 8616580, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 853925378, u'bytes_used': 1762586624, u'op_per_sec': 12, u'version': 1684, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2383032, u'write_bytes_sec': 1084200, u'bytes_avail': 427524427776}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:10.813 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:11.047 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:11.047 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:11.293 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:12.526 INFO:tasks.rados.rados.0.vpm137.stdout:3736: done (3 left) 2014-08-08T10:51:12.526 INFO:tasks.rados.rados.0.vpm137.stdout:3737: done (2 left) 2014-08-08T10:51:12.527 INFO:tasks.rados.rados.0.vpm137.stdout:3741: read oid 434 snap -1 2014-08-08T10:51:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:3742: write oid 367 current snap is 0 2014-08-08T10:51:12.529 INFO:tasks.rados.rados.0.vpm137.stdout:3742: seq_num 2292 ranges {799041=529634,1827916=171125} 2014-08-08T10:51:12.573 INFO:tasks.rados.rados.0.vpm137.stdout:3742: writing vpm1375878-367 from 799041 to 1328675 tid 1 2014-08-08T10:51:12.595 INFO:tasks.rados.rados.0.vpm137.stdout:3742: writing vpm1375878-367 from 1827916 to 1999041 tid 2 2014-08-08T10:51:12.597 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 1 to vpm1375878-236 2014-08-08T10:51:12.597 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 2 to vpm1375878-236 2014-08-08T10:51:12.597 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 3 to vpm1375878-236 2014-08-08T10:51:12.598 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 4 to vpm1375878-236 2014-08-08T10:51:12.598 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 6 to vpm1375878-236 2014-08-08T10:51:12.598 INFO:tasks.rados.rados.0.vpm137.stdout:3739: finishing write tid 7 to vpm1375878-236 2014-08-08T10:51:12.600 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 236 v 320 (ObjNum 2290 snap 0 seq_num 2290) dirty exists 2014-08-08T10:51:12.600 INFO:tasks.rados.rados.0.vpm137.stdout:3740: finishing write tid 1 to vpm1375878-346 2014-08-08T10:51:12.600 INFO:tasks.rados.rados.0.vpm137.stdout:3740: finishing write tid 2 to vpm1375878-346 2014-08-08T10:51:12.601 INFO:tasks.rados.rados.0.vpm137.stdout:3740: finishing write tid 4 to vpm1375878-346 2014-08-08T10:51:12.601 INFO:tasks.rados.rados.0.vpm137.stdout:3740: finishing write tid 5 to vpm1375878-346 2014-08-08T10:51:12.601 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 346 v 609 (ObjNum 2291 snap 0 seq_num 2291) dirty exists 2014-08-08T10:51:12.601 INFO:tasks.rados.rados.0.vpm137.stdout:3741: expect (ObjNum 2144 snap 0 seq_num 2144) 2014-08-08T10:51:13.022 INFO:tasks.rados.rados.0.vpm137.stdout:3739: done (3 left) 2014-08-08T10:51:13.022 INFO:tasks.rados.rados.0.vpm137.stdout:3740: done (2 left) 2014-08-08T10:51:13.023 INFO:tasks.rados.rados.0.vpm137.stdout:3741: done (1 left) 2014-08-08T10:51:13.024 INFO:tasks.rados.rados.0.vpm137.stdout:3743: read oid 156 snap -1 2014-08-08T10:51:13.024 INFO:tasks.rados.rados.0.vpm137.stdout:3744: write oid 62 current snap is 0 2014-08-08T10:51:13.024 INFO:tasks.rados.rados.0.vpm137.stdout:3744: seq_num 2293 ranges {495295=785087,1695294=1} 2014-08-08T10:51:13.070 INFO:tasks.rados.rados.0.vpm137.stdout:3744: writing vpm1375878-62 from 495295 to 1280382 tid 1 2014-08-08T10:51:13.076 INFO:tasks.rados.rados.0.vpm137.stdout:3744: writing vpm1375878-62 from 1695294 to 1695295 tid 2 2014-08-08T10:51:13.077 INFO:tasks.rados.rados.0.vpm137.stdout:3742: finishing write tid 1 to vpm1375878-367 2014-08-08T10:51:13.077 INFO:tasks.rados.rados.0.vpm137.stdout:3742: finishing write tid 2 to vpm1375878-367 2014-08-08T10:51:13.077 INFO:tasks.rados.rados.0.vpm137.stdout:3742: finishing write tid 4 to vpm1375878-367 2014-08-08T10:51:13.078 INFO:tasks.rados.rados.0.vpm137.stdout:3742: finishing write tid 5 to vpm1375878-367 2014-08-08T10:51:13.078 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 367 v 354 (ObjNum 2292 snap 0 seq_num 2292) dirty exists 2014-08-08T10:51:13.078 INFO:tasks.rados.rados.0.vpm137.stdout:3743: expect (ObjNum 2209 snap 0 seq_num 2209) 2014-08-08T10:51:13.113 INFO:tasks.rados.rados.0.vpm137.stdout:3742: done (2 left) 2014-08-08T10:51:13.114 INFO:tasks.rados.rados.0.vpm137.stdout:3743: done (1 left) 2014-08-08T10:51:13.114 INFO:tasks.rados.rados.0.vpm137.stdout:3745: read oid 130 snap -1 2014-08-08T10:51:13.115 INFO:tasks.rados.rados.0.vpm137.stdout:3746: read oid 144 snap -1 2014-08-08T10:51:13.116 INFO:tasks.rados.rados.0.vpm137.stdout:3747: write oid 292 current snap is 0 2014-08-08T10:51:13.116 INFO:tasks.rados.rados.0.vpm137.stdout:3747: seq_num 2294 ranges {590525=508315,1390524=1} 2014-08-08T10:51:13.176 INFO:tasks.rados.rados.0.vpm137.stdout:3747: writing vpm1375878-292 from 590525 to 1098840 tid 1 2014-08-08T10:51:13.182 INFO:tasks.rados.rados.0.vpm137.stdout:3747: writing vpm1375878-292 from 1390524 to 1390525 tid 2 2014-08-08T10:51:13.184 INFO:tasks.rados.rados.0.vpm137.stdout:3744: finishing write tid 1 to vpm1375878-62 2014-08-08T10:51:13.185 INFO:tasks.rados.rados.0.vpm137.stdout:3744: finishing write tid 2 to vpm1375878-62 2014-08-08T10:51:13.185 INFO:tasks.rados.rados.0.vpm137.stdout:3746: expect (ObjNum 2256 snap 0 seq_num 2256) 2014-08-08T10:51:14.321 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:14.537 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:14.582 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:15.922 INFO:tasks.rados.rados.0.vpm137.stdout:3746: done (3 left) 2014-08-08T10:51:15.922 INFO:tasks.rados.rados.0.vpm137.stdout:3748: delete oid 382 current snap is 0 2014-08-08T10:51:15.924 INFO:tasks.rados.rados.0.vpm137.stdout:3744: finishing write tid 4 to vpm1375878-62 2014-08-08T10:51:15.924 INFO:tasks.rados.rados.0.vpm137.stdout:3744: finishing write tid 5 to vpm1375878-62 2014-08-08T10:51:15.925 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 62 v 357 (ObjNum 2293 snap 0 seq_num 2293) dirty exists 2014-08-08T10:51:15.925 INFO:tasks.rados.rados.0.vpm137.stdout:3745: expect (ObjNum 763 snap 0 seq_num 763) 2014-08-08T10:51:16.440 INFO:tasks.rados.rados.0.vpm137.stdout:3744: done (3 left) 2014-08-08T10:51:16.440 INFO:tasks.rados.rados.0.vpm137.stdout:3745: done (2 left) 2014-08-08T10:51:16.441 INFO:tasks.rados.rados.0.vpm137.stdout:3748: done (1 left) 2014-08-08T10:51:16.441 INFO:tasks.rados.rados.0.vpm137.stdout:3749: read oid 358 snap -1 2014-08-08T10:51:16.442 INFO:tasks.rados.rados.0.vpm137.stdout:3750: delete oid 336 current snap is 0 2014-08-08T10:51:16.443 INFO:tasks.rados.rados.0.vpm137.stdout:3747: finishing write tid 1 to vpm1375878-292 2014-08-08T10:51:16.443 INFO:tasks.rados.rados.0.vpm137.stdout:3747: finishing write tid 2 to vpm1375878-292 2014-08-08T10:51:16.444 INFO:tasks.rados.rados.0.vpm137.stdout:3747: finishing write tid 4 to vpm1375878-292 2014-08-08T10:51:16.444 INFO:tasks.rados.rados.0.vpm137.stdout:3747: finishing write tid 5 to vpm1375878-292 2014-08-08T10:51:16.444 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 292 v 421 (ObjNum 2294 snap 0 seq_num 2294) dirty exists 2014-08-08T10:51:16.466 INFO:tasks.rados.rados.0.vpm137.stdout:3747: done (2 left) 2014-08-08T10:51:16.466 INFO:tasks.rados.rados.0.vpm137.stdout:3750: done (1 left) 2014-08-08T10:51:16.466 INFO:tasks.rados.rados.0.vpm137.stdout:3751: write oid 274 current snap is 0 2014-08-08T10:51:16.466 INFO:tasks.rados.rados.0.vpm137.stdout:3751: seq_num 2297 ranges {691766=708400,2058609=597880,3091765=1} 2014-08-08T10:51:16.494 INFO:tasks.rados.rados.0.vpm137.stdout:3751: writing vpm1375878-274 from 691766 to 1400166 tid 1 2014-08-08T10:51:16.531 INFO:tasks.rados.rados.0.vpm137.stdout:3751: writing vpm1375878-274 from 2058609 to 2656489 tid 2 2014-08-08T10:51:16.537 INFO:tasks.rados.rados.0.vpm137.stdout:3751: writing vpm1375878-274 from 3091765 to 3091766 tid 3 2014-08-08T10:51:16.538 INFO:tasks.rados.rados.0.vpm137.stdout:3749: expect (ObjNum 2114 snap 0 seq_num 2114) 2014-08-08T10:51:17.123 INFO:tasks.rados.rados.0.vpm137.stdout:3749: done (1 left) 2014-08-08T10:51:17.123 INFO:tasks.rados.rados.0.vpm137.stdout:3752: write oid 62 current snap is 0 2014-08-08T10:51:17.124 INFO:tasks.rados.rados.0.vpm137.stdout:3752: seq_num 2298 ranges {485172=400000} 2014-08-08T10:51:17.142 INFO:tasks.rados.rados.0.vpm137.stdout:3752: writing vpm1375878-62 from 485172 to 885172 tid 1 2014-08-08T10:51:17.143 INFO:tasks.rados.rados.0.vpm137.stdout:3751: finishing write tid 1 to vpm1375878-274 2014-08-08T10:51:17.143 INFO:tasks.rados.rados.0.vpm137.stdout:3751: finishing write tid 2 to vpm1375878-274 2014-08-08T10:51:17.144 INFO:tasks.rados.rados.0.vpm137.stdout:3751: finishing write tid 3 to vpm1375878-274 2014-08-08T10:51:17.145 INFO:tasks.rados.rados.0.vpm137.stdout:3751: finishing write tid 5 to vpm1375878-274 2014-08-08T10:51:17.145 INFO:tasks.rados.rados.0.vpm137.stdout:3751: finishing write tid 6 to vpm1375878-274 2014-08-08T10:51:17.145 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 274 v 357 (ObjNum 2297 snap 0 seq_num 2297) dirty exists 2014-08-08T10:51:17.145 INFO:tasks.rados.rados.0.vpm137.stdout:3751: done (1 left) 2014-08-08T10:51:17.146 INFO:tasks.rados.rados.0.vpm137.stdout:3753: read oid 489 snap -1 2014-08-08T10:51:17.146 INFO:tasks.rados.rados.0.vpm137.stdout:3754: read oid 141 snap -1 2014-08-08T10:51:17.146 INFO:tasks.rados.rados.0.vpm137.stdout:3755: write oid 349 current snap is 0 2014-08-08T10:51:17.147 INFO:tasks.rados.rados.0.vpm137.stdout:3755: seq_num 2299 ranges {579376=1} 2014-08-08T10:51:17.152 INFO:tasks.rados.rados.0.vpm137.stdout:3755: writing vpm1375878-349 from 579376 to 579377 tid 1 2014-08-08T10:51:17.153 INFO:tasks.rados.rados.0.vpm137.stdout:3756: write oid 306 current snap is 0 2014-08-08T10:51:17.154 INFO:tasks.rados.rados.0.vpm137.stdout:3756: seq_num 2300 ranges {791982=1} 2014-08-08T10:51:17.161 INFO:tasks.rados.rados.0.vpm137.stdout:3756: writing vpm1375878-306 from 791982 to 791983 tid 1 2014-08-08T10:51:17.162 INFO:tasks.rados.rados.0.vpm137.stdout:3752: finishing write tid 1 to vpm1375878-62 2014-08-08T10:51:17.163 INFO:tasks.rados.rados.0.vpm137.stdout:3757: read oid 236 snap -1 2014-08-08T10:51:17.163 INFO:tasks.rados.rados.0.vpm137.stdout:3758: write oid 291 current snap is 0 2014-08-08T10:51:17.164 INFO:tasks.rados.rados.0.vpm137.stdout:3758: seq_num 2301 ranges {584365=783137,1775169=630084,2584364=1} 2014-08-08T10:51:17.217 INFO:tasks.rados.rados.0.vpm137.stdout:3758: writing vpm1375878-291 from 584365 to 1367502 tid 1 2014-08-08T10:51:17.271 INFO:tasks.rados.rados.0.vpm137.stdout:3758: writing vpm1375878-291 from 1775169 to 2405253 tid 2 2014-08-08T10:51:17.281 INFO:tasks.rados.rados.0.vpm137.stdout:3758: writing vpm1375878-291 from 2584364 to 2584365 tid 3 2014-08-08T10:51:17.284 INFO:tasks.rados.rados.0.vpm137.stdout:3752: finishing write tid 3 to vpm1375878-62 2014-08-08T10:51:17.285 INFO:tasks.rados.rados.0.vpm137.stdout:3752: finishing write tid 4 to vpm1375878-62 2014-08-08T10:51:17.285 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 62 v 359 (ObjNum 2298 snap 0 seq_num 2298) dirty exists 2014-08-08T10:51:17.285 INFO:tasks.rados.rados.0.vpm137.stdout:3753: expect (ObjNum 993 snap 0 seq_num 993) 2014-08-08T10:51:17.715 INFO:teuthology.orchestra.run.vpm135.stderr:2014-08-08 17:51:17.715046 7f158430d700 0 -- :/1030947 >> 10.214.139.155:6789/0 pipe(0x7f158002fb30 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f158002fdc0).fault 2014-08-08T10:51:17.828 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 856492640, u'bytes_used': 1707274240, u'op_per_sec': 8, u'version': 1686, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1272189, u'write_bytes_sec': 481502, u'bytes_avail': 427579740160}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:17.829 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:17.887 INFO:tasks.rados.rados.0.vpm137.stdout:3752: done (6 left) 2014-08-08T10:51:17.887 INFO:tasks.rados.rados.0.vpm137.stdout:3753: done (5 left) 2014-08-08T10:51:17.887 INFO:tasks.rados.rados.0.vpm137.stdout:3759: write oid 130 current snap is 0 2014-08-08T10:51:17.887 INFO:tasks.rados.rados.0.vpm137.stdout:3759: seq_num 2302 ranges {680619=507390,1758103=451704,2280618=1} 2014-08-08T10:51:17.908 INFO:tasks.rados.rados.0.vpm137.stdout:3759: writing vpm1375878-130 from 680619 to 1188009 tid 1 2014-08-08T10:51:17.930 INFO:tasks.rados.rados.0.vpm137.stdout:3759: writing vpm1375878-130 from 1758103 to 2209807 tid 2 2014-08-08T10:51:17.932 INFO:tasks.rados.rados.0.vpm137.stdout:3759: writing vpm1375878-130 from 2280618 to 2280619 tid 3 2014-08-08T10:51:17.932 INFO:tasks.rados.rados.0.vpm137.stdout:3754: expect (ObjNum 2212 snap 0 seq_num 2212) 2014-08-08T10:51:18.088 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:18.088 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:18.296 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:18.630 INFO:tasks.rados.rados.0.vpm137.stdout:3754: done (5 left) 2014-08-08T10:51:18.631 INFO:tasks.rados.rados.0.vpm137.stdout:3760: delete oid 165 current snap is 0 2014-08-08T10:51:18.632 INFO:tasks.rados.rados.0.vpm137.stdout:3755: finishing write tid 1 to vpm1375878-349 2014-08-08T10:51:18.632 INFO:tasks.rados.rados.0.vpm137.stdout:3755: finishing write tid 3 to vpm1375878-349 2014-08-08T10:51:18.632 INFO:tasks.rados.rados.0.vpm137.stdout:3755: finishing write tid 4 to vpm1375878-349 2014-08-08T10:51:18.633 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 349 v 611 (ObjNum 2299 snap 0 seq_num 2299) dirty exists 2014-08-08T10:51:18.633 INFO:tasks.rados.rados.0.vpm137.stdout:3756: finishing write tid 1 to vpm1375878-306 2014-08-08T10:51:18.633 INFO:tasks.rados.rados.0.vpm137.stdout:3757: expect (ObjNum 2290 snap 0 seq_num 2290) 2014-08-08T10:51:21.213 INFO:tasks.rados.rados.0.vpm137.stdout:3755: done (5 left) 2014-08-08T10:51:21.213 INFO:tasks.rados.rados.0.vpm137.stdout:3757: done (4 left) 2014-08-08T10:51:21.214 INFO:tasks.rados.rados.0.vpm137.stdout:3760: done (3 left) 2014-08-08T10:51:21.214 INFO:tasks.rados.rados.0.vpm137.stdout:3761: write oid 17 current snap is 0 2014-08-08T10:51:21.215 INFO:tasks.rados.rados.0.vpm137.stdout:3761: seq_num 2304 ranges {285605=1} 2014-08-08T10:51:21.216 INFO:tasks.rados.rados.0.vpm137.stdout:3761: writing vpm1375878-17 from 285605 to 285606 tid 1 2014-08-08T10:51:21.217 INFO:tasks.rados.rados.0.vpm137.stdout:3762: read oid 253 snap -1 2014-08-08T10:51:21.219 INFO:tasks.rados.rados.0.vpm137.stdout:3756: finishing write tid 3 to vpm1375878-306 2014-08-08T10:51:21.219 INFO:tasks.rados.rados.0.vpm137.stdout:3756: finishing write tid 4 to vpm1375878-306 2014-08-08T10:51:21.219 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 306 v 509 (ObjNum 2300 snap 0 seq_num 2300) dirty exists 2014-08-08T10:51:21.219 INFO:tasks.rados.rados.0.vpm137.stdout:3758: finishing write tid 1 to vpm1375878-291 2014-08-08T10:51:21.220 INFO:tasks.rados.rados.0.vpm137.stdout:3758: finishing write tid 2 to vpm1375878-291 2014-08-08T10:51:21.220 INFO:tasks.rados.rados.0.vpm137.stdout:3758: finishing write tid 3 to vpm1375878-291 2014-08-08T10:51:21.220 INFO:tasks.rados.rados.0.vpm137.stdout:3758: finishing write tid 5 to vpm1375878-291 2014-08-08T10:51:21.220 INFO:tasks.rados.rados.0.vpm137.stdout:3758: finishing write tid 6 to vpm1375878-291 2014-08-08T10:51:21.220 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 291 v 363 (ObjNum 2301 snap 0 seq_num 2301) dirty exists 2014-08-08T10:51:21.221 INFO:tasks.rados.rados.0.vpm137.stdout:3759: finishing write tid 1 to vpm1375878-130 2014-08-08T10:51:21.221 INFO:tasks.rados.rados.0.vpm137.stdout:3759: finishing write tid 2 to vpm1375878-130 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3759: finishing write tid 3 to vpm1375878-130 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3759: finishing write tid 5 to vpm1375878-130 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3759: finishing write tid 6 to vpm1375878-130 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 130 v 325 (ObjNum 2302 snap 0 seq_num 2302) dirty exists 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3756: done (4 left) 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3758: done (3 left) 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3759: done (2 left) 2014-08-08T10:51:21.222 INFO:tasks.rados.rados.0.vpm137.stdout:3763: read oid 13 snap -1 2014-08-08T10:51:21.223 INFO:tasks.rados.rados.0.vpm137.stdout:3764: read oid 496 snap -1 2014-08-08T10:51:21.223 INFO:tasks.rados.rados.0.vpm137.stdout:3765: read oid 119 snap -1 2014-08-08T10:51:21.223 INFO:tasks.rados.rados.0.vpm137.stdout:3766: write oid 148 current snap is 0 2014-08-08T10:51:21.223 INFO:tasks.rados.rados.0.vpm137.stdout:3766: seq_num 2305 ranges {781860=458899,1912979=697361,3182473=700283,3981859=1} 2014-08-08T10:51:21.273 INFO:tasks.rados.rados.0.vpm137.stdout:3766: writing vpm1375878-148 from 781860 to 1240759 tid 1 2014-08-08T10:51:21.326 INFO:tasks.rados.rados.0.vpm137.stdout:3766: writing vpm1375878-148 from 1912979 to 2610340 tid 2 2014-08-08T10:51:21.328 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:21.364 INFO:tasks.rados.rados.0.vpm137.stdout:3766: writing vpm1375878-148 from 3182473 to 3882756 tid 3 2014-08-08T10:51:21.368 INFO:tasks.rados.rados.0.vpm137.stdout:3766: writing vpm1375878-148 from 3981859 to 3981860 tid 4 2014-08-08T10:51:21.369 INFO:tasks.rados.rados.0.vpm137.stdout:3762: expect (ObjNum 2260 snap 0 seq_num 2260) 2014-08-08T10:51:21.564 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:21.614 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:22.191 INFO:tasks.rados.rados.0.vpm137.stdout:3762: done (5 left) 2014-08-08T10:51:22.192 INFO:tasks.rados.rados.0.vpm137.stdout:3767: write oid 284 current snap is 0 2014-08-08T10:51:22.192 INFO:tasks.rados.rados.0.vpm137.stdout:3767: seq_num 2306 ranges {477089=582127,1626086=767556,2984721=587074,3677088=1} 2014-08-08T10:51:22.225 INFO:tasks.rados.rados.0.vpm137.stdout:3767: writing vpm1375878-284 from 477089 to 1059216 tid 1 2014-08-08T10:51:22.277 INFO:tasks.rados.rados.0.vpm137.stdout:3767: writing vpm1375878-284 from 1626086 to 2393642 tid 2 2014-08-08T10:51:22.318 INFO:tasks.rados.rados.0.vpm137.stdout:3767: writing vpm1375878-284 from 2984721 to 3571795 tid 3 2014-08-08T10:51:22.320 INFO:tasks.rados.rados.0.vpm137.stdout:3767: writing vpm1375878-284 from 3677088 to 3677089 tid 4 2014-08-08T10:51:22.321 INFO:tasks.rados.rados.0.vpm137.stdout:3763: expect (ObjNum 1896 snap 0 seq_num 1896) 2014-08-08T10:51:24.504 INFO:tasks.rados.rados.0.vpm137.stdout:3763: done (5 left) 2014-08-08T10:51:24.504 INFO:tasks.rados.rados.0.vpm137.stdout:3768: delete oid 218 current snap is 0 2014-08-08T10:51:24.505 INFO:tasks.rados.rados.0.vpm137.stdout:3764: expect (ObjNum 2072 snap 0 seq_num 2072) 2014-08-08T10:51:24.731 INFO:teuthology.orchestra.run.vpm135.stderr:2014-08-08 17:51:24.730826 7fc100206700 0 -- :/1031080 >> 10.214.139.155:6789/0 pipe(0x7fc0fc02fb30 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7fc0fc02fdc0).fault 2014-08-08T10:51:24.854 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 850789997, u'bytes_used': 1705185280, u'op_per_sec': 8, u'version': 1688, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1429056, u'write_bytes_sec': 648036, u'bytes_avail': 427581829120}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:24.855 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:25.058 INFO:tasks.rados.rados.0.vpm137.stdout:3764: done (5 left) 2014-08-08T10:51:25.058 INFO:tasks.rados.rados.0.vpm137.stdout:3768: done (4 left) 2014-08-08T10:51:25.058 INFO:tasks.rados.rados.0.vpm137.stdout:3769: read oid 140 snap -1 2014-08-08T10:51:25.059 INFO:tasks.rados.rados.0.vpm137.stdout:3770: write oid 427 current snap is 0 2014-08-08T10:51:25.059 INFO:tasks.rados.rados.0.vpm137.stdout:3770: seq_num 2308 ranges {765725=400000} 2014-08-08T10:51:25.094 INFO:tasks.rados.rados.0.vpm137.stdout:3770: writing vpm1375878-427 from 765725 to 1165725 tid 1 2014-08-08T10:51:25.101 INFO:tasks.rados.rados.0.vpm137.stdout:3761: finishing write tid 1 to vpm1375878-17 2014-08-08T10:51:25.102 INFO:tasks.rados.rados.0.vpm137.stdout:3761: finishing write tid 3 to vpm1375878-17 2014-08-08T10:51:25.102 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 1 to vpm1375878-148 2014-08-08T10:51:25.102 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 2 to vpm1375878-148 2014-08-08T10:51:25.103 INFO:tasks.rados.rados.0.vpm137.stdout:3761: finishing write tid 4 to vpm1375878-17 2014-08-08T10:51:25.103 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 17 v 423 (ObjNum 2304 snap 0 seq_num 2304) dirty exists 2014-08-08T10:51:25.103 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 3 to vpm1375878-148 2014-08-08T10:51:25.103 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 4 to vpm1375878-148 2014-08-08T10:51:25.104 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 6 to vpm1375878-148 2014-08-08T10:51:25.104 INFO:tasks.rados.rados.0.vpm137.stdout:3765: expect (ObjNum 1739 snap 0 seq_num 1739) 2014-08-08T10:51:26.418 INFO:tasks.rados.rados.0.vpm137.stdout:3761: done (5 left) 2014-08-08T10:51:26.418 INFO:tasks.rados.rados.0.vpm137.stdout:3765: done (4 left) 2014-08-08T10:51:26.418 INFO:tasks.rados.rados.0.vpm137.stdout:3771: read oid 315 snap -1 2014-08-08T10:51:26.418 INFO:tasks.rados.rados.0.vpm137.stdout:3772: delete oid 333 current snap is 0 2014-08-08T10:51:26.419 INFO:tasks.rados.rados.0.vpm137.stdout:3766: finishing write tid 7 to vpm1375878-148 2014-08-08T10:51:26.419 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 148 v 442 (ObjNum 2305 snap 0 seq_num 2305) dirty exists 2014-08-08T10:51:26.419 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 1 to vpm1375878-284 2014-08-08T10:51:26.419 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 2 to vpm1375878-284 2014-08-08T10:51:26.420 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 3 to vpm1375878-284 2014-08-08T10:51:26.420 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 4 to vpm1375878-284 2014-08-08T10:51:26.420 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 6 to vpm1375878-284 2014-08-08T10:51:26.420 INFO:tasks.rados.rados.0.vpm137.stdout:3767: finishing write tid 7 to vpm1375878-284 2014-08-08T10:51:26.420 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 284 v 357 (ObjNum 2306 snap 0 seq_num 2306) dirty exists 2014-08-08T10:51:26.421 INFO:tasks.rados.rados.0.vpm137.stdout:3769: expect (ObjNum 2195 snap 0 seq_num 2195) 2014-08-08T10:51:26.605 INFO:tasks.rados.rados.0.vpm137.stdout:3770: finishing write tid 1 to vpm1375878-427 2014-08-08T10:51:26.605 INFO:tasks.rados.rados.0.vpm137.stdout:3770: finishing write tid 3 to vpm1375878-427 2014-08-08T10:51:26.605 INFO:tasks.rados.rados.0.vpm137.stdout:3770: finishing write tid 4 to vpm1375878-427 2014-08-08T10:51:26.605 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 427 v 371 (ObjNum 2308 snap 0 seq_num 2308) dirty exists 2014-08-08T10:51:26.622 INFO:tasks.rados.rados.0.vpm137.stdout:3766: done (5 left) 2014-08-08T10:51:26.623 INFO:tasks.rados.rados.0.vpm137.stdout:3767: done (4 left) 2014-08-08T10:51:26.623 INFO:tasks.rados.rados.0.vpm137.stdout:3769: done (3 left) 2014-08-08T10:51:26.624 INFO:tasks.rados.rados.0.vpm137.stdout:3770: done (2 left) 2014-08-08T10:51:26.624 INFO:tasks.rados.rados.0.vpm137.stdout:3772: done (1 left) 2014-08-08T10:51:26.624 INFO:tasks.rados.rados.0.vpm137.stdout:3773: write oid 468 current snap is 0 2014-08-08T10:51:26.624 INFO:tasks.rados.rados.0.vpm137.stdout:3773: seq_num 2310 ranges {522136=505440,1460022=551185,2122135=1} 2014-08-08T10:51:26.643 INFO:tasks.rados.rados.0.vpm137.stdout:3773: writing vpm1375878-468 from 522136 to 1027576 tid 1 2014-08-08T10:51:26.668 INFO:tasks.rados.rados.0.vpm137.stdout:3773: writing vpm1375878-468 from 1460022 to 2011207 tid 2 2014-08-08T10:51:26.670 INFO:tasks.rados.rados.0.vpm137.stdout:3773: writing vpm1375878-468 from 2122135 to 2122136 tid 3 2014-08-08T10:51:26.671 INFO:tasks.rados.rados.0.vpm137.stdout:3774: delete oid 401 current snap is 0 2014-08-08T10:51:26.760 INFO:tasks.rados.rados.0.vpm137.stdout:3773: finishing write tid 1 to vpm1375878-468 2014-08-08T10:51:26.766 INFO:tasks.rados.rados.0.vpm137.stdout:3773: finishing write tid 2 to vpm1375878-468 2014-08-08T10:51:26.766 INFO:tasks.rados.rados.0.vpm137.stdout:3773: finishing write tid 3 to vpm1375878-468 2014-08-08T10:51:26.766 INFO:tasks.rados.rados.0.vpm137.stdout:3773: finishing write tid 5 to vpm1375878-468 2014-08-08T10:51:26.767 INFO:tasks.rados.rados.0.vpm137.stdout:3774: done (2 left) 2014-08-08T10:51:26.767 INFO:tasks.rados.rados.0.vpm137.stdout:3775: write oid 242 current snap is 0 2014-08-08T10:51:26.767 INFO:tasks.rados.rados.0.vpm137.stdout:3775: seq_num 2312 ranges {562196=600473,1651167=691575,2562195=1} 2014-08-08T10:51:26.794 INFO:tasks.rados.rados.0.vpm137.stdout:3775: writing vpm1375878-242 from 562196 to 1162669 tid 1 2014-08-08T10:51:26.833 INFO:tasks.rados.rados.0.vpm137.stdout:3775: writing vpm1375878-242 from 1651167 to 2342742 tid 2 2014-08-08T10:51:26.838 INFO:tasks.rados.rados.0.vpm137.stdout:3775: writing vpm1375878-242 from 2562195 to 2562196 tid 3 2014-08-08T10:51:26.839 INFO:tasks.rados.rados.0.vpm137.stdout:3773: finishing write tid 6 to vpm1375878-468 2014-08-08T10:51:26.839 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 468 v 421 (ObjNum 2310 snap 0 seq_num 2310) dirty exists 2014-08-08T10:51:26.839 INFO:tasks.rados.rados.0.vpm137.stdout:3771: expect (ObjNum 1888 snap 0 seq_num 1888) 2014-08-08T10:51:27.731 INFO:teuthology.orchestra.run.vpm135.stderr:2014-08-08 17:51:27.730922 7f7540772700 0 -- :/1031113 >> 10.214.139.155:6789/0 pipe(0x7f753c02fb30 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f753c02fdc0).fault 2014-08-08T10:51:28.121 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:28.121 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:28.344 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:28.942 INFO:tasks.rados.rados.0.vpm137.stdout:3771: done (2 left) 2014-08-08T10:51:28.942 INFO:tasks.rados.rados.0.vpm137.stdout:3773: done (1 left) 2014-08-08T10:51:28.942 INFO:tasks.rados.rados.0.vpm137.stdout:3776: read oid 325 snap -1 2014-08-08T10:51:28.943 INFO:tasks.rados.rados.0.vpm137.stdout:3777: write oid 206 current snap is 0 2014-08-08T10:51:28.943 INFO:tasks.rados.rados.0.vpm137.stdout:3777: seq_num 2313 ranges {623378=457973,1615923=512171,2702931=632367,3823377=1} 2014-08-08T10:51:28.962 INFO:tasks.rados.rados.0.vpm137.stdout:3777: writing vpm1375878-206 from 623378 to 1081351 tid 1 2014-08-08T10:51:28.987 INFO:tasks.rados.rados.0.vpm137.stdout:3777: writing vpm1375878-206 from 1615923 to 2128094 tid 2 2014-08-08T10:51:29.012 INFO:tasks.rados.rados.0.vpm137.stdout:3777: writing vpm1375878-206 from 2702931 to 3335298 tid 3 2014-08-08T10:51:29.018 INFO:tasks.rados.rados.0.vpm137.stdout:3777: writing vpm1375878-206 from 3823377 to 3823378 tid 4 2014-08-08T10:51:29.018 INFO:tasks.rados.rados.0.vpm137.stdout:3775: finishing write tid 1 to vpm1375878-242 2014-08-08T10:51:29.019 INFO:tasks.rados.rados.0.vpm137.stdout:3775: finishing write tid 2 to vpm1375878-242 2014-08-08T10:51:29.019 INFO:tasks.rados.rados.0.vpm137.stdout:3775: finishing write tid 3 to vpm1375878-242 2014-08-08T10:51:29.019 INFO:tasks.rados.rados.0.vpm137.stdout:3775: finishing write tid 5 to vpm1375878-242 2014-08-08T10:51:29.020 INFO:tasks.rados.rados.0.vpm137.stdout:3775: finishing write tid 6 to vpm1375878-242 2014-08-08T10:51:29.020 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 242 v 329 (ObjNum 2312 snap 0 seq_num 2312) dirty exists 2014-08-08T10:51:29.020 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 1 to vpm1375878-206 2014-08-08T10:51:29.020 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 2 to vpm1375878-206 2014-08-08T10:51:29.020 INFO:tasks.rados.rados.0.vpm137.stdout:3775: done (2 left) 2014-08-08T10:51:29.021 INFO:tasks.rados.rados.0.vpm137.stdout:3776: done (1 left) 2014-08-08T10:51:29.021 INFO:tasks.rados.rados.0.vpm137.stdout:3778: write oid 180 current snap is 0 2014-08-08T10:51:29.021 INFO:tasks.rados.rados.0.vpm137.stdout:3778: seq_num 2314 ranges {567182=1} 2014-08-08T10:51:29.024 INFO:tasks.rados.rados.0.vpm137.stdout:3778: writing vpm1375878-180 from 567182 to 567183 tid 1 2014-08-08T10:51:29.024 INFO:tasks.rados.rados.0.vpm137.stdout:3779: read oid 407 snap -1 2014-08-08T10:51:29.025 INFO:tasks.rados.rados.0.vpm137.stdout:3780: write oid 18 current snap is 0 2014-08-08T10:51:29.025 INFO:tasks.rados.rados.0.vpm137.stdout:3780: seq_num 2315 ranges {262412=1} 2014-08-08T10:51:29.027 INFO:tasks.rados.rados.0.vpm137.stdout:3780: writing vpm1375878-18 from 262412 to 262413 tid 1 2014-08-08T10:51:29.028 INFO:tasks.rados.rados.0.vpm137.stdout:3781: read oid 224 snap -1 2014-08-08T10:51:29.028 INFO:tasks.rados.rados.0.vpm137.stdout:3782: write oid 197 current snap is 0 2014-08-08T10:51:29.028 INFO:tasks.rados.rados.0.vpm137.stdout:3782: seq_num 2316 ranges {607243=400000} 2014-08-08T10:51:29.045 INFO:tasks.rados.rados.0.vpm137.stdout:3782: writing vpm1375878-197 from 607243 to 1007243 tid 1 2014-08-08T10:51:29.048 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 3 to vpm1375878-206 2014-08-08T10:51:29.048 INFO:tasks.rados.rados.0.vpm137.stdout:3783: write oid 262 current snap is 0 2014-08-08T10:51:29.048 INFO:tasks.rados.rados.0.vpm137.stdout:3783: seq_num 2317 ranges {551049=532711,1483908=267141} 2014-08-08T10:51:29.069 INFO:tasks.rados.rados.0.vpm137.stdout:3783: writing vpm1375878-262 from 551049 to 1083760 tid 1 2014-08-08T10:51:29.081 INFO:tasks.rados.rados.0.vpm137.stdout:3783: writing vpm1375878-262 from 1483908 to 1751049 tid 2 2014-08-08T10:51:29.082 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 4 to vpm1375878-206 2014-08-08T10:51:29.082 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 6 to vpm1375878-206 2014-08-08T10:51:29.082 INFO:tasks.rados.rados.0.vpm137.stdout:3777: finishing write tid 7 to vpm1375878-206 2014-08-08T10:51:29.083 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 206 v 617 (ObjNum 2313 snap 0 seq_num 2313) dirty exists 2014-08-08T10:51:29.083 INFO:tasks.rados.rados.0.vpm137.stdout:3777: done (6 left) 2014-08-08T10:51:29.083 INFO:tasks.rados.rados.0.vpm137.stdout:3784: read oid 33 snap -1 2014-08-08T10:51:29.083 INFO:tasks.rados.rados.0.vpm137.stdout:3785: delete oid 213 current snap is 0 2014-08-08T10:51:29.107 INFO:tasks.rados.rados.0.vpm137.stdout:3778: finishing write tid 1 to vpm1375878-180 2014-08-08T10:51:29.147 INFO:tasks.rados.rados.0.vpm137.stdout:3781: expect (ObjNum 2096 snap 0 seq_num 2096) 2014-08-08T10:51:29.835 INFO:tasks.rados.rados.0.vpm137.stdout:3781: done (7 left) 2014-08-08T10:51:29.835 INFO:tasks.rados.rados.0.vpm137.stdout:3785: done (6 left) 2014-08-08T10:51:29.836 INFO:tasks.rados.rados.0.vpm137.stdout:3786: write oid 280 current snap is 0 2014-08-08T10:51:29.836 INFO:tasks.rados.rados.0.vpm137.stdout:3786: seq_num 2319 ranges {707460=476320,1773229=719838,2707459=1} 2014-08-08T10:51:29.857 INFO:tasks.rados.rados.0.vpm137.stdout:3786: writing vpm1375878-280 from 707460 to 1183780 tid 1 2014-08-08T10:51:29.891 INFO:tasks.rados.rados.0.vpm137.stdout:3786: writing vpm1375878-280 from 1773229 to 2493067 tid 2 2014-08-08T10:51:29.896 INFO:tasks.rados.rados.0.vpm137.stdout:3786: writing vpm1375878-280 from 2707459 to 2707460 tid 3 2014-08-08T10:51:29.897 INFO:tasks.rados.rados.0.vpm137.stdout:3778: finishing write tid 3 to vpm1375878-180 2014-08-08T10:51:29.897 INFO:tasks.rados.rados.0.vpm137.stdout:3780: finishing write tid 1 to vpm1375878-18 2014-08-08T10:51:29.898 INFO:tasks.rados.rados.0.vpm137.stdout:3780: finishing write tid 3 to vpm1375878-18 2014-08-08T10:51:29.898 INFO:tasks.rados.rados.0.vpm137.stdout:3778: finishing write tid 4 to vpm1375878-180 2014-08-08T10:51:29.899 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 180 v 315 (ObjNum 2314 snap 0 seq_num 2314) dirty exists 2014-08-08T10:51:29.899 INFO:tasks.rados.rados.0.vpm137.stdout:3780: finishing write tid 4 to vpm1375878-18 2014-08-08T10:51:29.899 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 18 v 423 (ObjNum 2315 snap 0 seq_num 2315) dirty exists 2014-08-08T10:51:29.899 INFO:tasks.rados.rados.0.vpm137.stdout:3782: finishing write tid 1 to vpm1375878-197 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:3782: finishing write tid 3 to vpm1375878-197 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:3783: finishing write tid 1 to vpm1375878-262 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:3783: finishing write tid 2 to vpm1375878-262 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:3783: finishing write tid 4 to vpm1375878-262 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:3782: finishing write tid 4 to vpm1375878-197 2014-08-08T10:51:29.900 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 197 v 425 (ObjNum 2316 snap 0 seq_num 2316) dirty exists 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:3783: finishing write tid 5 to vpm1375878-262 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 262 v 442 (ObjNum 2317 snap 0 seq_num 2317) dirty exists 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:3778: done (6 left) 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:3780: done (5 left) 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:3782: done (4 left) 2014-08-08T10:51:29.901 INFO:tasks.rados.rados.0.vpm137.stdout:3783: done (3 left) 2014-08-08T10:51:29.902 INFO:tasks.rados.rados.0.vpm137.stdout:3784: done (2 left) 2014-08-08T10:51:29.902 INFO:tasks.rados.rados.0.vpm137.stdout:3787: read oid 225 snap -1 2014-08-08T10:51:29.902 INFO:tasks.rados.rados.0.vpm137.stdout:3788: read oid 397 snap -1 2014-08-08T10:51:29.902 INFO:tasks.rados.rados.0.vpm137.stdout:3789: write oid 289 current snap is 0 2014-08-08T10:51:29.903 INFO:tasks.rados.rados.0.vpm137.stdout:3789: seq_num 2320 ranges {403714=600572,1755137=648577} 2014-08-08T10:51:29.928 INFO:tasks.rados.rados.0.vpm137.stdout:3789: writing vpm1375878-289 from 403714 to 1004286 tid 1 2014-08-08T10:51:29.968 INFO:tasks.rados.rados.0.vpm137.stdout:3789: writing vpm1375878-289 from 1755137 to 2403714 tid 2 2014-08-08T10:51:29.971 INFO:tasks.rados.rados.0.vpm137.stdout:3786: finishing write tid 1 to vpm1375878-280 2014-08-08T10:51:29.971 INFO:tasks.rados.rados.0.vpm137.stdout:3790: delete oid 448 current snap is 0 2014-08-08T10:51:30.004 INFO:tasks.rados.rados.0.vpm137.stdout:3779: expect (ObjNum 1594 snap 0 seq_num 1594) 2014-08-08T10:51:31.373 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:31.565 INFO:tasks.rados.rados.0.vpm137.stdout:3779: done (5 left) 2014-08-08T10:51:31.566 INFO:tasks.rados.rados.0.vpm137.stdout:3790: done (4 left) 2014-08-08T10:51:31.566 INFO:tasks.rados.rados.0.vpm137.stdout:3791: write oid 461 current snap is 0 2014-08-08T10:51:31.566 INFO:tasks.rados.rados.0.vpm137.stdout:3791: seq_num 2322 ranges {408700=1} 2014-08-08T10:51:31.569 INFO:tasks.rados.rados.0.vpm137.stdout:3791: writing vpm1375878-461 from 408700 to 408701 tid 1 2014-08-08T10:51:31.570 INFO:tasks.rados.rados.0.vpm137.stdout:3792: write oid 310 current snap is 0 2014-08-08T10:51:31.570 INFO:tasks.rados.rados.0.vpm137.stdout:3792: seq_num 2323 ranges {104954=1} 2014-08-08T10:51:31.571 INFO:tasks.rados.rados.0.vpm137.stdout:3792: writing vpm1375878-310 from 104954 to 104955 tid 1 2014-08-08T10:51:31.572 INFO:tasks.rados.rados.0.vpm137.stdout:3788: expect (ObjNum 2137 snap 0 seq_num 2137) 2014-08-08T10:51:32.174 INFO:tasks.rados.rados.0.vpm137.stdout:3788: done (5 left) 2014-08-08T10:51:32.174 INFO:tasks.rados.rados.0.vpm137.stdout:3793: write oid 302 current snap is 0 2014-08-08T10:51:32.175 INFO:tasks.rados.rados.0.vpm137.stdout:3793: seq_num 2324 ranges {696313=675309,1896312=1} 2014-08-08T10:51:32.201 INFO:tasks.rados.rados.0.vpm137.stdout:3793: writing vpm1375878-302 from 696313 to 1371622 tid 1 2014-08-08T10:51:32.215 INFO:tasks.rados.rados.0.vpm137.stdout:3793: writing vpm1375878-302 from 1896312 to 1896313 tid 2 2014-08-08T10:51:32.215 INFO:tasks.rados.rados.0.vpm137.stdout:3787: expect (ObjNum 2117 snap 0 seq_num 2117) 2014-08-08T10:51:33.970 INFO:tasks.rados.rados.0.vpm137.stdout:3787: done (5 left) 2014-08-08T10:51:33.971 INFO:tasks.rados.rados.0.vpm137.stdout:3794: write oid 192 current snap is 0 2014-08-08T10:51:33.971 INFO:tasks.rados.rados.0.vpm137.stdout:3794: seq_num 2325 ranges {792566=530762,1592565=1} 2014-08-08T10:51:33.994 INFO:tasks.rados.rados.0.vpm137.stdout:3794: writing vpm1375878-192 from 792566 to 1323328 tid 1 2014-08-08T10:51:33.999 INFO:tasks.rados.rados.0.vpm137.stdout:3794: writing vpm1375878-192 from 1592565 to 1592566 tid 2 2014-08-08T10:51:34.000 INFO:tasks.rados.rados.0.vpm137.stdout:3786: finishing write tid 2 to vpm1375878-280 2014-08-08T10:51:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3786: finishing write tid 3 to vpm1375878-280 2014-08-08T10:51:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3786: finishing write tid 5 to vpm1375878-280 2014-08-08T10:51:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3789: finishing write tid 1 to vpm1375878-289 2014-08-08T10:51:34.001 INFO:tasks.rados.rados.0.vpm137.stdout:3786: finishing write tid 6 to vpm1375878-280 2014-08-08T10:51:34.002 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 280 v 446 (ObjNum 2319 snap 0 seq_num 2319) dirty exists 2014-08-08T10:51:34.002 INFO:tasks.rados.rados.0.vpm137.stdout:3789: finishing write tid 2 to vpm1375878-289 2014-08-08T10:51:34.017 INFO:tasks.rados.rados.0.vpm137.stdout:3789: finishing write tid 4 to vpm1375878-289 2014-08-08T10:51:34.017 INFO:tasks.rados.rados.0.vpm137.stdout:3789: finishing write tid 5 to vpm1375878-289 2014-08-08T10:51:34.018 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 289 v 449 (ObjNum 2320 snap 0 seq_num 2320) dirty exists 2014-08-08T10:51:34.018 INFO:tasks.rados.rados.0.vpm137.stdout:3791: finishing write tid 1 to vpm1375878-461 2014-08-08T10:51:34.018 INFO:tasks.rados.rados.0.vpm137.stdout:3791: finishing write tid 3 to vpm1375878-461 2014-08-08T10:51:34.018 INFO:tasks.rados.rados.0.vpm137.stdout:3792: finishing write tid 1 to vpm1375878-310 2014-08-08T10:51:34.019 INFO:tasks.rados.rados.0.vpm137.stdout:3792: finishing write tid 3 to vpm1375878-310 2014-08-08T10:51:34.019 INFO:tasks.rados.rados.0.vpm137.stdout:3791: finishing write tid 4 to vpm1375878-461 2014-08-08T10:51:34.019 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 461 v 430 (ObjNum 2322 snap 0 seq_num 2322) dirty exists 2014-08-08T10:51:34.019 INFO:tasks.rados.rados.0.vpm137.stdout:3792: finishing write tid 4 to vpm1375878-310 2014-08-08T10:51:34.019 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 310 v 332 (ObjNum 2323 snap 0 seq_num 2323) dirty exists 2014-08-08T10:51:34.020 INFO:tasks.rados.rados.0.vpm137.stdout:3793: finishing write tid 1 to vpm1375878-302 2014-08-08T10:51:34.020 INFO:tasks.rados.rados.0.vpm137.stdout:3793: finishing write tid 2 to vpm1375878-302 2014-08-08T10:51:34.020 INFO:tasks.rados.rados.0.vpm137.stdout:3793: finishing write tid 4 to vpm1375878-302 2014-08-08T10:51:34.020 INFO:tasks.rados.rados.0.vpm137.stdout:3793: finishing write tid 5 to vpm1375878-302 2014-08-08T10:51:34.021 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 302 v 620 (ObjNum 2324 snap 0 seq_num 2324) dirty exists 2014-08-08T10:51:34.021 INFO:tasks.rados.rados.0.vpm137.stdout:3786: done (5 left) 2014-08-08T10:51:34.021 INFO:tasks.rados.rados.0.vpm137.stdout:3789: done (4 left) 2014-08-08T10:51:34.021 INFO:tasks.rados.rados.0.vpm137.stdout:3791: done (3 left) 2014-08-08T10:51:34.021 INFO:tasks.rados.rados.0.vpm137.stdout:3792: done (2 left) 2014-08-08T10:51:34.022 INFO:tasks.rados.rados.0.vpm137.stdout:3793: done (1 left) 2014-08-08T10:51:34.022 INFO:tasks.rados.rados.0.vpm137.stdout:3795: delete oid 295 current snap is 0 2014-08-08T10:51:34.173 INFO:tasks.rados.rados.0.vpm137.stdout:3794: finishing write tid 1 to vpm1375878-192 2014-08-08T10:51:34.174 INFO:tasks.rados.rados.0.vpm137.stdout:3794: finishing write tid 2 to vpm1375878-192 2014-08-08T10:51:34.176 INFO:tasks.rados.rados.0.vpm137.stdout:3794: finishing write tid 4 to vpm1375878-192 2014-08-08T10:51:34.177 INFO:tasks.rados.rados.0.vpm137.stdout:3795: done (1 left) 2014-08-08T10:51:34.177 INFO:tasks.rados.rados.0.vpm137.stdout:3796: write oid 235 current snap is 0 2014-08-08T10:51:34.177 INFO:tasks.rados.rados.0.vpm137.stdout:3796: seq_num 2327 ranges {797554=626818,1992525=419319,3144790=452764} 2014-08-08T10:51:34.217 INFO:tasks.rados.rados.0.vpm137.stdout:3796: writing vpm1375878-235 from 797554 to 1424372 tid 1 2014-08-08T10:51:34.244 INFO:tasks.rados.rados.0.vpm137.stdout:3796: writing vpm1375878-235 from 1992525 to 2411844 tid 2 2014-08-08T10:51:34.267 INFO:tasks.rados.rados.0.vpm137.stdout:3796: writing vpm1375878-235 from 3144790 to 3597554 tid 3 2014-08-08T10:51:34.272 INFO:tasks.rados.rados.0.vpm137.stdout:3797: write oid 363 current snap is 0 2014-08-08T10:51:34.272 INFO:tasks.rados.rados.0.vpm137.stdout:3797: seq_num 2328 ranges {493808=750047,1857058=489515,2944993=348815} 2014-08-08T10:51:34.313 INFO:tasks.rados.rados.0.vpm137.stdout:3797: writing vpm1375878-363 from 493808 to 1243855 tid 1 2014-08-08T10:51:34.344 INFO:tasks.rados.rados.0.vpm137.stdout:3797: writing vpm1375878-363 from 1857058 to 2346573 tid 2 2014-08-08T10:51:34.369 INFO:tasks.rados.rados.0.vpm137.stdout:3797: writing vpm1375878-363 from 2944993 to 3293808 tid 3 2014-08-08T10:51:34.371 INFO:tasks.rados.rados.0.vpm137.stdout:3798: read oid 212 snap -1 2014-08-08T10:51:34.371 INFO:tasks.rados.rados.0.vpm137.stdout:3799: read oid 303 snap -1 2014-08-08T10:51:34.372 INFO:tasks.rados.rados.0.vpm137.stdout:3800: write oid 186 current snap is 0 2014-08-08T10:51:34.372 INFO:tasks.rados.rados.0.vpm137.stdout:3800: seq_num 2329 ranges {589037=454075,1702389=559710,2877417=111620} 2014-08-08T10:51:34.410 INFO:tasks.rados.rados.0.vpm137.stdout:3800: writing vpm1375878-186 from 589037 to 1043112 tid 1 2014-08-08T10:51:34.451 INFO:tasks.rados.rados.0.vpm137.stdout:3800: writing vpm1375878-186 from 1702389 to 2262099 tid 2 2014-08-08T10:51:34.452 INFO:teuthology.orchestra.run.vpm135.stderr:2014-08-08 17:51:34.450886 7f8e987e9700 0 -- :/1031181 >> 10.214.139.155:6789/0 pipe(0x7f8e9402fb60 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f8e9402fdf0).fault 2014-08-08T10:51:34.466 INFO:tasks.rados.rados.0.vpm137.stdout:3800: writing vpm1375878-186 from 2877417 to 2989037 tid 3 2014-08-08T10:51:34.468 INFO:tasks.rados.rados.0.vpm137.stdout:3796: finishing write tid 1 to vpm1375878-235 2014-08-08T10:51:34.468 INFO:tasks.rados.rados.0.vpm137.stdout:3798: expect (ObjNum 2249 snap 0 seq_num 2249) 2014-08-08T10:51:34.556 INFO:tasks.rados.rados.0.vpm137.stdout:3798: done (5 left) 2014-08-08T10:51:34.556 INFO:tasks.rados.rados.0.vpm137.stdout:3801: write oid 381 current snap is 0 2014-08-08T10:51:34.556 INFO:tasks.rados.rados.0.vpm137.stdout:3801: seq_num 2330 ranges {533867=578327,1533897=781329,2831090=545502,3733866=1} 2014-08-08T10:51:34.581 INFO:tasks.rados.rados.0.vpm137.stdout:3801: writing vpm1375878-381 from 533867 to 1112194 tid 1 2014-08-08T10:51:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:3801: writing vpm1375878-381 from 1533897 to 2315226 tid 2 2014-08-08T10:51:34.642 INFO:tasks.rados.rados.0.vpm137.stdout:3801: writing vpm1375878-381 from 2831090 to 3376592 tid 3 2014-08-08T10:51:34.647 INFO:tasks.rados.rados.0.vpm137.stdout:3801: writing vpm1375878-381 from 3733866 to 3733867 tid 4 2014-08-08T10:51:34.648 INFO:tasks.rados.rados.0.vpm137.stdout:3794: finishing write tid 5 to vpm1375878-192 2014-08-08T10:51:34.648 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 192 v 466 (ObjNum 2325 snap 0 seq_num 2325) dirty exists 2014-08-08T10:51:34.648 INFO:tasks.rados.rados.0.vpm137.stdout:3796: finishing write tid 2 to vpm1375878-235 2014-08-08T10:51:34.648 INFO:tasks.rados.rados.0.vpm137.stdout:3796: finishing write tid 3 to vpm1375878-235 2014-08-08T10:51:34.649 INFO:tasks.rados.rados.0.vpm137.stdout:3796: finishing write tid 5 to vpm1375878-235 2014-08-08T10:51:34.649 INFO:tasks.rados.rados.0.vpm137.stdout:3797: finishing write tid 1 to vpm1375878-363 2014-08-08T10:51:34.649 INFO:tasks.rados.rados.0.vpm137.stdout:3797: finishing write tid 2 to vpm1375878-363 2014-08-08T10:51:34.649 INFO:tasks.rados.rados.0.vpm137.stdout:3797: finishing write tid 3 to vpm1375878-363 2014-08-08T10:51:34.649 INFO:tasks.rados.rados.0.vpm137.stdout:3796: finishing write tid 6 to vpm1375878-235 2014-08-08T10:51:34.650 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 235 v 434 (ObjNum 2327 snap 0 seq_num 2327) dirty exists 2014-08-08T10:51:34.650 INFO:tasks.rados.rados.0.vpm137.stdout:3797: finishing write tid 5 to vpm1375878-363 2014-08-08T10:51:34.650 INFO:tasks.rados.rados.0.vpm137.stdout:3800: finishing write tid 1 to vpm1375878-186 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3800: finishing write tid 2 to vpm1375878-186 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3800: finishing write tid 3 to vpm1375878-186 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3800: finishing write tid 5 to vpm1375878-186 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3794: done (5 left) 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3796: done (4 left) 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3799: done (3 left) 2014-08-08T10:51:34.651 INFO:tasks.rados.rados.0.vpm137.stdout:3802: write oid 491 current snap is 0 2014-08-08T10:51:34.652 INFO:tasks.rados.rados.0.vpm137.stdout:3802: seq_num 2331 ranges {595049=400000} 2014-08-08T10:51:34.670 INFO:tasks.rados.rados.0.vpm137.stdout:3802: writing vpm1375878-491 from 595049 to 995049 tid 1 2014-08-08T10:51:34.672 INFO:tasks.rados.rados.0.vpm137.stdout:3797: finishing write tid 6 to vpm1375878-363 2014-08-08T10:51:34.673 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 363 v 427 (ObjNum 2328 snap 0 seq_num 2328) dirty exists 2014-08-08T10:51:34.673 INFO:tasks.rados.rados.0.vpm137.stdout:3797: done (3 left) 2014-08-08T10:51:34.673 INFO:tasks.rados.rados.0.vpm137.stdout:3803: write oid 482 current snap is 0 2014-08-08T10:51:34.673 INFO:tasks.rados.rados.0.vpm137.stdout:3803: seq_num 2332 ranges {689254=1} 2014-08-08T10:51:34.677 INFO:tasks.rados.rados.0.vpm137.stdout:3803: writing vpm1375878-482 from 689254 to 689255 tid 1 2014-08-08T10:51:34.678 INFO:tasks.rados.rados.0.vpm137.stdout:3804: write oid 376 current snap is 0 2014-08-08T10:51:34.679 INFO:tasks.rados.rados.0.vpm137.stdout:3804: seq_num 2333 ranges {634084=528812,1434083=1} 2014-08-08T10:51:34.701 INFO:tasks.rados.rados.0.vpm137.stdout:3804: writing vpm1375878-376 from 634084 to 1162896 tid 1 2014-08-08T10:51:34.706 INFO:tasks.rados.rados.0.vpm137.stdout:3804: writing vpm1375878-376 from 1434083 to 1434084 tid 2 2014-08-08T10:51:34.706 INFO:tasks.rados.rados.0.vpm137.stdout:3805: write oid 428 current snap is 0 2014-08-08T10:51:34.706 INFO:tasks.rados.rados.0.vpm137.stdout:3805: seq_num 2334 ranges {577890=653065,1650458=527432} 2014-08-08T10:51:34.736 INFO:tasks.rados.rados.0.vpm137.stdout:3805: writing vpm1375878-428 from 577890 to 1230955 tid 1 2014-08-08T10:51:34.747 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:34.759 INFO:tasks.rados.rados.0.vpm137.stdout:3805: writing vpm1375878-428 from 1650458 to 2177890 tid 2 2014-08-08T10:51:34.763 INFO:tasks.rados.rados.0.vpm137.stdout:3800: finishing write tid 6 to vpm1375878-186 2014-08-08T10:51:34.764 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 186 v 414 (ObjNum 2329 snap 0 seq_num 2329) dirty exists 2014-08-08T10:51:34.764 INFO:tasks.rados.rados.0.vpm137.stdout:3800: done (5 left) 2014-08-08T10:51:34.764 INFO:tasks.rados.rados.0.vpm137.stdout:3806: write oid 303 current snap is 0 2014-08-08T10:51:34.764 INFO:tasks.rados.rados.0.vpm137.stdout:3806: seq_num 2335 ranges {674144=624869,1874143=1} 2014-08-08T10:51:34.801 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:34.802 INFO:tasks.rados.rados.0.vpm137.stdout:3806: writing vpm1375878-303 from 674144 to 1299013 tid 1 2014-08-08T10:51:34.802 INFO:tasks.rados.rados.0.vpm137.stdout:3806: writing vpm1375878-303 from 1874143 to 1874144 tid 2 2014-08-08T10:51:34.802 INFO:tasks.rados.rados.0.vpm137.stdout:3807: write oid 487 current snap is 0 2014-08-08T10:51:34.802 INFO:tasks.rados.rados.0.vpm137.stdout:3807: seq_num 2336 ranges {735326=748097,2110403=435524,3135325=1} 2014-08-08T10:51:34.827 INFO:tasks.rados.rados.0.vpm137.stdout:3807: writing vpm1375878-487 from 735326 to 1483423 tid 1 2014-08-08T10:51:34.848 INFO:tasks.rados.rados.0.vpm137.stdout:3807: writing vpm1375878-487 from 2110403 to 2545927 tid 2 2014-08-08T10:51:34.853 INFO:tasks.rados.rados.0.vpm137.stdout:3807: writing vpm1375878-487 from 3135325 to 3135326 tid 3 2014-08-08T10:51:34.853 INFO:tasks.rados.rados.0.vpm137.stdout:3808: write oid 141 current snap is 0 2014-08-08T10:51:34.854 INFO:tasks.rados.rados.0.vpm137.stdout:3808: seq_num 2337 ranges {679131=604574,1921687=657143,3080500=590796,3879130=1} 2014-08-08T10:51:34.880 INFO:tasks.rados.rados.0.vpm137.stdout:3808: writing vpm1375878-141 from 679131 to 1283705 tid 1 2014-08-08T10:51:34.952 INFO:tasks.rados.rados.0.vpm137.stdout:3808: writing vpm1375878-141 from 1921687 to 2578830 tid 2 2014-08-08T10:51:34.979 INFO:tasks.rados.rados.0.vpm137.stdout:3808: writing vpm1375878-141 from 3080500 to 3671296 tid 3 2014-08-08T10:51:34.982 INFO:tasks.rados.rados.0.vpm137.stdout:3808: writing vpm1375878-141 from 3879130 to 3879131 tid 4 2014-08-08T10:51:34.983 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 1 to vpm1375878-381 2014-08-08T10:51:34.983 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 2 to vpm1375878-381 2014-08-08T10:51:34.983 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 3 to vpm1375878-381 2014-08-08T10:51:34.984 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 4 to vpm1375878-381 2014-08-08T10:51:34.984 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 6 to vpm1375878-381 2014-08-08T10:51:34.984 INFO:tasks.rados.rados.0.vpm137.stdout:3802: finishing write tid 1 to vpm1375878-491 2014-08-08T10:51:34.984 INFO:tasks.rados.rados.0.vpm137.stdout:3809: delete oid 393 current snap is 0 2014-08-08T10:51:34.995 INFO:tasks.rados.rados.0.vpm137.stdout:3802: finishing write tid 3 to vpm1375878-491 2014-08-08T10:51:34.995 INFO:tasks.rados.rados.0.vpm137.stdout:3803: finishing write tid 1 to vpm1375878-482 2014-08-08T10:51:35.020 INFO:tasks.rados.rados.0.vpm137.stdout:3803: finishing write tid 3 to vpm1375878-482 2014-08-08T10:51:35.021 INFO:tasks.rados.rados.0.vpm137.stdout:3804: finishing write tid 1 to vpm1375878-376 2014-08-08T10:51:35.021 INFO:tasks.rados.rados.0.vpm137.stdout:3801: finishing write tid 7 to vpm1375878-381 2014-08-08T10:51:35.021 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 381 v 454 (ObjNum 2330 snap 0 seq_num 2330) dirty exists 2014-08-08T10:51:35.073 INFO:tasks.rados.rados.0.vpm137.stdout:3804: finishing write tid 2 to vpm1375878-376 2014-08-08T10:51:35.073 INFO:tasks.rados.rados.0.vpm137.stdout:3804: finishing write tid 4 to vpm1375878-376 2014-08-08T10:51:35.073 INFO:tasks.rados.rados.0.vpm137.stdout:3806: finishing write tid 1 to vpm1375878-303 2014-08-08T10:51:35.074 INFO:tasks.rados.rados.0.vpm137.stdout:3805: finishing write tid 1 to vpm1375878-428 2014-08-08T10:51:35.096 INFO:tasks.rados.rados.0.vpm137.stdout:3805: finishing write tid 2 to vpm1375878-428 2014-08-08T10:51:35.097 INFO:tasks.rados.rados.0.vpm137.stdout:3802: finishing write tid 4 to vpm1375878-491 2014-08-08T10:51:35.097 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 491 v 511 (ObjNum 2331 snap 0 seq_num 2331) dirty exists 2014-08-08T10:51:35.098 INFO:tasks.rados.rados.0.vpm137.stdout:3803: finishing write tid 4 to vpm1375878-482 2014-08-08T10:51:35.098 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 482 v 401 (ObjNum 2332 snap 0 seq_num 2332) dirty exists 2014-08-08T10:51:35.099 INFO:tasks.rados.rados.0.vpm137.stdout:3805: finishing write tid 4 to vpm1375878-428 2014-08-08T10:51:35.099 INFO:tasks.rados.rados.0.vpm137.stdout:3806: finishing write tid 2 to vpm1375878-303 2014-08-08T10:51:35.139 INFO:tasks.rados.rados.0.vpm137.stdout:3806: finishing write tid 4 to vpm1375878-303 2014-08-08T10:51:35.140 INFO:tasks.rados.rados.0.vpm137.stdout:3807: finishing write tid 1 to vpm1375878-487 2014-08-08T10:51:35.141 INFO:tasks.rados.rados.0.vpm137.stdout:3807: finishing write tid 2 to vpm1375878-487 2014-08-08T10:51:35.141 INFO:tasks.rados.rados.0.vpm137.stdout:3807: finishing write tid 3 to vpm1375878-487 2014-08-08T10:51:35.173 INFO:tasks.rados.rados.0.vpm137.stdout:3807: finishing write tid 5 to vpm1375878-487 2014-08-08T10:51:35.174 INFO:tasks.rados.rados.0.vpm137.stdout:3804: finishing write tid 5 to vpm1375878-376 2014-08-08T10:51:35.174 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 376 v 417 (ObjNum 2333 snap 0 seq_num 2333) dirty exists 2014-08-08T10:51:35.174 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 1 to vpm1375878-141 2014-08-08T10:51:35.188 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 2 to vpm1375878-141 2014-08-08T10:51:35.217 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 3 to vpm1375878-141 2014-08-08T10:51:35.224 INFO:tasks.rados.rados.0.vpm137.stdout:3805: finishing write tid 5 to vpm1375878-428 2014-08-08T10:51:35.225 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 428 v 404 (ObjNum 2334 snap 0 seq_num 2334) dirty exists 2014-08-08T10:51:35.241 INFO:tasks.rados.rados.0.vpm137.stdout:3806: finishing write tid 5 to vpm1375878-303 2014-08-08T10:51:35.242 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 303 v 420 (ObjNum 2335 snap 0 seq_num 2335) dirty exists 2014-08-08T10:51:35.269 INFO:tasks.rados.rados.0.vpm137.stdout:3807: finishing write tid 6 to vpm1375878-487 2014-08-08T10:51:35.269 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 487 v 367 (ObjNum 2336 snap 0 seq_num 2336) dirty exists 2014-08-08T10:51:35.321 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 849713387, u'bytes_used': 1707139072, u'op_per_sec': 15, u'version': 1690, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2859795, u'write_bytes_sec': 1357514, u'bytes_avail': 427579875328}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:35.321 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:35.438 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 4 to vpm1375878-141 2014-08-08T10:51:35.446 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 6 to vpm1375878-141 2014-08-08T10:51:35.446 INFO:tasks.rados.rados.0.vpm137.stdout:3808: finishing write tid 7 to vpm1375878-141 2014-08-08T10:51:35.446 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 141 v 425 (ObjNum 2337 snap 0 seq_num 2337) dirty exists 2014-08-08T10:51:35.448 INFO:tasks.rados.rados.0.vpm137.stdout:3801: done (8 left) 2014-08-08T10:51:35.448 INFO:tasks.rados.rados.0.vpm137.stdout:3802: done (7 left) 2014-08-08T10:51:35.449 INFO:tasks.rados.rados.0.vpm137.stdout:3803: done (6 left) 2014-08-08T10:51:35.449 INFO:tasks.rados.rados.0.vpm137.stdout:3804: done (5 left) 2014-08-08T10:51:35.449 INFO:tasks.rados.rados.0.vpm137.stdout:3805: done (4 left) 2014-08-08T10:51:35.449 INFO:tasks.rados.rados.0.vpm137.stdout:3806: done (3 left) 2014-08-08T10:51:35.450 INFO:tasks.rados.rados.0.vpm137.stdout:3807: done (2 left) 2014-08-08T10:51:35.450 INFO:tasks.rados.rados.0.vpm137.stdout:3808: done (1 left) 2014-08-08T10:51:35.450 INFO:tasks.rados.rados.0.vpm137.stdout:3809: done (0 left) 2014-08-08T10:51:35.450 INFO:tasks.rados.rados.0.vpm137.stdout:3810: delete oid 440 current snap is 0 2014-08-08T10:51:35.473 INFO:tasks.rados.rados.0.vpm137.stdout:3810: done (0 left) 2014-08-08T10:51:35.473 INFO:tasks.rados.rados.0.vpm137.stdout:3811: read oid 341 snap -1 2014-08-08T10:51:35.473 INFO:tasks.rados.rados.0.vpm137.stdout:3812: read oid 239 snap -1 2014-08-08T10:51:35.474 INFO:tasks.rados.rados.0.vpm137.stdout:3813: write oid 323 current snap is 0 2014-08-08T10:51:35.474 INFO:tasks.rados.rados.0.vpm137.stdout:3813: seq_num 2340 ranges {779349=671411,1579348=1} 2014-08-08T10:51:35.503 INFO:tasks.rados.rados.0.vpm137.stdout:3813: writing vpm1375878-323 from 779349 to 1450760 tid 1 2014-08-08T10:51:35.507 INFO:tasks.rados.rados.0.vpm137.stdout:3813: writing vpm1375878-323 from 1579348 to 1579349 tid 2 2014-08-08T10:51:35.508 INFO:tasks.rados.rados.0.vpm137.stdout:3814: read oid 385 snap -1 2014-08-08T10:51:35.509 INFO:tasks.rados.rados.0.vpm137.stdout:3815: read oid 408 snap -1 2014-08-08T10:51:35.509 INFO:tasks.rados.rados.0.vpm137.stdout:3816: write oid 283 current snap is 0 2014-08-08T10:51:35.509 INFO:tasks.rados.rados.0.vpm137.stdout:3816: seq_num 2341 ranges {475602=527887,1275601=1} 2014-08-08T10:51:35.531 INFO:tasks.rados.rados.0.vpm137.stdout:3816: writing vpm1375878-283 from 475602 to 1003489 tid 1 2014-08-08T10:51:35.535 INFO:tasks.rados.rados.0.vpm137.stdout:3816: writing vpm1375878-283 from 1275601 to 1275602 tid 2 2014-08-08T10:51:35.535 INFO:tasks.rados.rados.0.vpm137.stdout:3817: write oid 108 current snap is 0 2014-08-08T10:51:35.535 INFO:tasks.rados.rados.0.vpm137.stdout:3817: seq_num 2342 ranges {419408=652139,1621179=398229} 2014-08-08T10:51:35.561 INFO:tasks.rados.rados.0.vpm137.stdout:3817: writing vpm1375878-108 from 419408 to 1071547 tid 1 2014-08-08T10:51:35.579 INFO:tasks.rados.rados.0.vpm137.stdout:3817: writing vpm1375878-108 from 1621179 to 2019408 tid 2 2014-08-08T10:51:35.580 INFO:tasks.rados.rados.0.vpm137.stdout:3818: read oid 203 snap -1 2014-08-08T10:51:35.582 INFO:tasks.rados.rados.0.vpm137.stdout:3819: write oid 326 current snap is 0 2014-08-08T10:51:35.582 INFO:tasks.rados.rados.0.vpm137.stdout:3819: seq_num 2343 ranges {515662=623944,1582864=132798} 2014-08-08T10:51:35.606 INFO:tasks.rados.rados.0.vpm137.stdout:3819: writing vpm1375878-326 from 515662 to 1139606 tid 1 2014-08-08T10:51:35.614 INFO:tasks.rados.rados.0.vpm137.stdout:3819: writing vpm1375878-326 from 1582864 to 1715662 tid 2 2014-08-08T10:51:35.615 INFO:tasks.rados.rados.0.vpm137.stdout:3820: write oid 357 current snap is 0 2014-08-08T10:51:35.615 INFO:tasks.rados.rados.0.vpm137.stdout:3820: seq_num 2344 ranges {459468=748196,1696996=535006,2459467=1} 2014-08-08T10:51:35.643 INFO:tasks.rados.rados.0.vpm137.stdout:3820: writing vpm1375878-357 from 459468 to 1207664 tid 1 2014-08-08T10:51:35.668 INFO:tasks.rados.rados.0.vpm137.stdout:3820: writing vpm1375878-357 from 1696996 to 2232002 tid 2 2014-08-08T10:51:35.671 INFO:tasks.rados.rados.0.vpm137.stdout:3820: writing vpm1375878-357 from 2459467 to 2459468 tid 3 2014-08-08T10:51:35.672 INFO:tasks.rados.rados.0.vpm137.stdout:3821: write oid 363 current snap is 0 2014-08-08T10:51:35.672 INFO:tasks.rados.rados.0.vpm137.stdout:3821: seq_num 2345 ranges {520649=603648,1776055=756625,3072127=555904,3720648=1} 2014-08-08T10:51:35.694 INFO:tasks.rados.rados.0.vpm137.stdout:3821: writing vpm1375878-363 from 520649 to 1124297 tid 1 2014-08-08T10:51:35.700 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:35.700 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:35.723 INFO:tasks.rados.rados.0.vpm137.stdout:3821: writing vpm1375878-363 from 1776055 to 2532680 tid 2 2014-08-08T10:51:35.747 INFO:tasks.rados.rados.0.vpm137.stdout:3821: writing vpm1375878-363 from 3072127 to 3628031 tid 3 2014-08-08T10:51:35.749 INFO:tasks.rados.rados.0.vpm137.stdout:3821: writing vpm1375878-363 from 3720648 to 3720649 tid 4 2014-08-08T10:51:35.750 INFO:tasks.rados.rados.0.vpm137.stdout:3822: read oid 483 snap -1 2014-08-08T10:51:35.750 INFO:tasks.rados.rados.0.vpm137.stdout:3823: write oid 472 current snap is 0 2014-08-08T10:51:35.750 INFO:tasks.rados.rados.0.vpm137.stdout:3823: seq_num 2346 ranges {616903=726877,1890188=426820,2873353=538823,3416902=1} 2014-08-08T10:51:35.783 INFO:tasks.rados.rados.0.vpm137.stdout:3823: writing vpm1375878-472 from 616903 to 1343780 tid 1 2014-08-08T10:51:35.816 INFO:tasks.rados.rados.0.vpm137.stdout:3823: writing vpm1375878-472 from 1890188 to 2317008 tid 2 2014-08-08T10:51:35.844 INFO:tasks.rados.rados.0.vpm137.stdout:3823: writing vpm1375878-472 from 2873353 to 3412176 tid 3 2014-08-08T10:51:35.846 INFO:tasks.rados.rados.0.vpm137.stdout:3823: writing vpm1375878-472 from 3416902 to 3416903 tid 4 2014-08-08T10:51:35.848 INFO:tasks.rados.rados.0.vpm137.stdout:3824: write oid 249 current snap is 0 2014-08-08T10:51:35.848 INFO:tasks.rados.rados.0.vpm137.stdout:3824: seq_num 2347 ranges {160708=1} 2014-08-08T10:51:35.850 INFO:tasks.rados.rados.0.vpm137.stdout:3824: writing vpm1375878-249 from 160708 to 160709 tid 1 2014-08-08T10:51:35.851 INFO:tasks.rados.rados.0.vpm137.stdout:3825: read oid 422 snap -1 2014-08-08T10:51:35.851 INFO:tasks.rados.rados.0.vpm137.stdout:3826: write oid 448 current snap is 0 2014-08-08T10:51:35.851 INFO:tasks.rados.rados.0.vpm137.stdout:3826: seq_num 2348 ranges {504514=400000} 2014-08-08T10:51:35.869 INFO:tasks.rados.rados.0.vpm137.stdout:3826: writing vpm1375878-448 from 504514 to 904514 tid 1 2014-08-08T10:51:35.871 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:36.005 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:36.238 INFO:tasks.rados.rados.0.vpm137.stdout:3815: expect (ObjNum 2175 snap 0 seq_num 2175) 2014-08-08T10:51:36.285 INFO:tasks.rados.rados.0.vpm137.stdout:3815: done (15 left) 2014-08-08T10:51:36.285 INFO:tasks.rados.rados.0.vpm137.stdout:3827: read oid 251 snap -1 2014-08-08T10:51:36.285 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:36.286 INFO:tasks.rados.rados.0.vpm137.stdout:3813: finishing write tid 1 to vpm1375878-323 2014-08-08T10:51:36.286 INFO:tasks.rados.rados.0.vpm137.stdout:3813: finishing write tid 2 to vpm1375878-323 2014-08-08T10:51:36.287 INFO:tasks.rados.rados.0.vpm137.stdout:3813: finishing write tid 4 to vpm1375878-323 2014-08-08T10:51:36.287 INFO:tasks.rados.rados.0.vpm137.stdout:3816: finishing write tid 1 to vpm1375878-283 2014-08-08T10:51:36.287 INFO:tasks.rados.rados.0.vpm137.stdout:3816: finishing write tid 2 to vpm1375878-283 2014-08-08T10:51:36.287 INFO:tasks.rados.rados.0.vpm137.stdout:3816: finishing write tid 4 to vpm1375878-283 2014-08-08T10:51:36.287 INFO:tasks.rados.rados.0.vpm137.stdout:3820: finishing write tid 1 to vpm1375878-357 2014-08-08T10:51:36.291 INFO:tasks.rados.rados.0.vpm137.stdout:3820: finishing write tid 2 to vpm1375878-357 2014-08-08T10:51:36.293 INFO:tasks.rados.rados.0.vpm137.stdout:3820: finishing write tid 3 to vpm1375878-357 2014-08-08T10:51:36.293 INFO:tasks.rados.rados.0.vpm137.stdout:3820: finishing write tid 5 to vpm1375878-357 2014-08-08T10:51:36.294 INFO:tasks.rados.rados.0.vpm137.stdout:3813: finishing write tid 5 to vpm1375878-323 2014-08-08T10:51:36.294 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 323 v 361 (ObjNum 2340 snap 0 seq_num 2340) dirty exists 2014-08-08T10:51:36.294 INFO:tasks.rados.rados.0.vpm137.stdout:3813: done (15 left) 2014-08-08T10:51:36.295 INFO:tasks.rados.rados.0.vpm137.stdout:3828: write oid 74 current snap is 0 2014-08-08T10:51:36.295 INFO:tasks.rados.rados.0.vpm137.stdout:3828: seq_num 2349 ranges {716096=400000} 2014-08-08T10:51:36.777 INFO:tasks.rados.rados.0.vpm137.stdout:3828: writing vpm1375878-74 from 716096 to 1116096 tid 1 2014-08-08T10:51:36.782 INFO:tasks.rados.rados.0.vpm137.stdout:3816: finishing write tid 5 to vpm1375878-283 2014-08-08T10:51:36.782 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 283 v 623 (ObjNum 2341 snap 0 seq_num 2341) dirty exists 2014-08-08T10:51:36.782 INFO:tasks.rados.rados.0.vpm137.stdout:3824: finishing write tid 1 to vpm1375878-249 2014-08-08T10:51:36.782 INFO:tasks.rados.rados.0.vpm137.stdout:3826: finishing write tid 1 to vpm1375878-448 2014-08-08T10:51:36.783 INFO:tasks.rados.rados.0.vpm137.stdout:3824: finishing write tid 3 to vpm1375878-249 2014-08-08T10:51:36.783 INFO:tasks.rados.rados.0.vpm137.stdout:3826: finishing write tid 3 to vpm1375878-448 2014-08-08T10:51:36.784 INFO:tasks.rados.rados.0.vpm137.stdout:3824: finishing write tid 4 to vpm1375878-249 2014-08-08T10:51:36.784 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 249 v 625 (ObjNum 2347 snap 0 seq_num 2347) dirty exists 2014-08-08T10:51:36.784 INFO:tasks.rados.rados.0.vpm137.stdout:3820: finishing write tid 6 to vpm1375878-357 2014-08-08T10:51:36.784 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 357 v 429 (ObjNum 2344 snap 0 seq_num 2344) dirty exists 2014-08-08T10:51:36.784 INFO:tasks.rados.rados.0.vpm137.stdout:3827: expect (ObjNum 2229 snap 0 seq_num 2229) 2014-08-08T10:51:37.984 INFO:tasks.rados.rados.0.vpm137.stdout:3816: done (15 left) 2014-08-08T10:51:37.984 INFO:tasks.rados.rados.0.vpm137.stdout:3820: done (14 left) 2014-08-08T10:51:37.985 INFO:tasks.rados.rados.0.vpm137.stdout:3824: done (13 left) 2014-08-08T10:51:37.985 INFO:tasks.rados.rados.0.vpm137.stdout:3827: done (12 left) 2014-08-08T10:51:37.985 INFO:tasks.rados.rados.0.vpm137.stdout:3829: delete oid 31 current snap is 0 2014-08-08T10:51:37.986 INFO:tasks.rados.rados.0.vpm137.stdout:3826: finishing write tid 4 to vpm1375878-448 2014-08-08T10:51:37.987 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 448 v 335 (ObjNum 2348 snap 0 seq_num 2348) dirty exists 2014-08-08T10:51:37.987 INFO:tasks.rados.rados.0.vpm137.stdout:3818: expect (ObjNum 1706 snap 0 seq_num 1706) 2014-08-08T10:51:39.036 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:39.326 INFO:tasks.rados.rados.0.vpm137.stdout:3818: done (12 left) 2014-08-08T10:51:39.326 INFO:tasks.rados.rados.0.vpm137.stdout:3826: done (11 left) 2014-08-08T10:51:39.326 INFO:tasks.rados.rados.0.vpm137.stdout:3829: done (10 left) 2014-08-08T10:51:39.326 INFO:tasks.rados.rados.0.vpm137.stdout:3830: write oid 201 current snap is 0 2014-08-08T10:51:39.327 INFO:tasks.rados.rados.0.vpm137.stdout:3830: seq_num 2351 ranges {605756=621995,1933361=564292,2605755=1} 2014-08-08T10:51:39.351 INFO:tasks.rados.rados.0.vpm137.stdout:3830: writing vpm1375878-201 from 605756 to 1227751 tid 1 2014-08-08T10:51:39.379 INFO:tasks.rados.rados.0.vpm137.stdout:3830: writing vpm1375878-201 from 1933361 to 2497653 tid 2 2014-08-08T10:51:39.382 INFO:tasks.rados.rados.0.vpm137.stdout:3830: writing vpm1375878-201 from 2605755 to 2605756 tid 3 2014-08-08T10:51:39.383 INFO:tasks.rados.rados.0.vpm137.stdout:3811: expect (ObjNum 340 snap 0 seq_num 340) 2014-08-08T10:51:40.365 INFO:tasks.rados.rados.0.vpm137.stdout:3811: done (10 left) 2014-08-08T10:51:40.365 INFO:tasks.rados.rados.0.vpm137.stdout:3831: write oid 333 current snap is 0 2014-08-08T10:51:40.365 INFO:tasks.rados.rados.0.vpm137.stdout:3831: seq_num 2352 ranges {700985=746247,2163844=137141} 2014-08-08T10:51:40.393 INFO:tasks.rados.rados.0.vpm137.stdout:3831: writing vpm1375878-333 from 700985 to 1447232 tid 1 2014-08-08T10:51:40.405 INFO:tasks.rados.rados.0.vpm137.stdout:3831: writing vpm1375878-333 from 2163844 to 2300985 tid 2 2014-08-08T10:51:40.406 INFO:tasks.rados.rados.0.vpm137.stdout:3812: expect (ObjNum 1976 snap 0 seq_num 1976) 2014-08-08T10:51:42.264 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:42.505 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:42.507 INFO:tasks.rados.rados.0.vpm137.stdout:3812: done (10 left) 2014-08-08T10:51:42.507 INFO:tasks.rados.rados.0.vpm137.stdout:3832: write oid 355 current snap is 0 2014-08-08T10:51:42.508 INFO:tasks.rados.rados.0.vpm137.stdout:3832: seq_num 2353 ranges {645815=718051,2126553=703658,3045814=1} 2014-08-08T10:51:42.549 INFO:tasks.rados.rados.0.vpm137.stdout:3832: writing vpm1375878-355 from 645815 to 1363866 tid 1 2014-08-08T10:51:42.596 INFO:tasks.rados.rados.0.vpm137.stdout:3832: writing vpm1375878-355 from 2126553 to 2830211 tid 2 2014-08-08T10:51:42.600 INFO:tasks.rados.rados.0.vpm137.stdout:3832: writing vpm1375878-355 from 3045814 to 3045815 tid 3 2014-08-08T10:51:42.601 INFO:tasks.rados.rados.0.vpm137.stdout:3814: expect (ObjNum 1524 snap 0 seq_num 1524) 2014-08-08T10:51:42.906 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 852637620, u'bytes_used': 1703669760, u'op_per_sec': 19, u'version': 1692, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1951263, u'write_bytes_sec': 2385649, u'bytes_avail': 427583344640}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:42.907 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:43.258 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:43.258 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:43.498 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:43.711 INFO:tasks.rados.rados.0.vpm137.stdout:3814: done (10 left) 2014-08-08T10:51:43.711 INFO:tasks.rados.rados.0.vpm137.stdout:3833: write oid 413 current snap is 0 2014-08-08T10:51:43.711 INFO:tasks.rados.rados.0.vpm137.stdout:3833: seq_num 2354 ranges {305972=1} 2014-08-08T10:51:43.714 INFO:tasks.rados.rados.0.vpm137.stdout:3833: writing vpm1375878-413 from 305972 to 305973 tid 1 2014-08-08T10:51:43.716 INFO:tasks.rados.rados.0.vpm137.stdout:3834: write oid 36 current snap is 0 2014-08-08T10:51:43.716 INFO:tasks.rados.rados.0.vpm137.stdout:3834: seq_num 2355 ranges {2226=1} 2014-08-08T10:51:43.717 INFO:tasks.rados.rados.0.vpm137.stdout:3834: writing vpm1375878-36 from 2226 to 2227 tid 1 2014-08-08T10:51:43.718 INFO:tasks.rados.rados.0.vpm137.stdout:3817: finishing write tid 1 to vpm1375878-108 2014-08-08T10:51:43.718 INFO:tasks.rados.rados.0.vpm137.stdout:3817: finishing write tid 2 to vpm1375878-108 2014-08-08T10:51:43.718 INFO:tasks.rados.rados.0.vpm137.stdout:3817: finishing write tid 4 to vpm1375878-108 2014-08-08T10:51:43.718 INFO:tasks.rados.rados.0.vpm137.stdout:3817: finishing write tid 5 to vpm1375878-108 2014-08-08T10:51:43.719 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 108 v 407 (ObjNum 2342 snap 0 seq_num 2342) dirty exists 2014-08-08T10:51:43.719 INFO:tasks.rados.rados.0.vpm137.stdout:3819: finishing write tid 1 to vpm1375878-326 2014-08-08T10:51:43.719 INFO:tasks.rados.rados.0.vpm137.stdout:3819: finishing write tid 2 to vpm1375878-326 2014-08-08T10:51:43.720 INFO:tasks.rados.rados.0.vpm137.stdout:3819: finishing write tid 4 to vpm1375878-326 2014-08-08T10:51:43.720 INFO:tasks.rados.rados.0.vpm137.stdout:3819: finishing write tid 5 to vpm1375878-326 2014-08-08T10:51:43.720 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 326 v 430 (ObjNum 2343 snap 0 seq_num 2343) dirty exists 2014-08-08T10:51:43.721 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 1 to vpm1375878-363 2014-08-08T10:51:43.721 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 2 to vpm1375878-363 2014-08-08T10:51:43.721 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 3 to vpm1375878-363 2014-08-08T10:51:43.721 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 4 to vpm1375878-363 2014-08-08T10:51:43.724 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 6 to vpm1375878-363 2014-08-08T10:51:43.724 INFO:tasks.rados.rados.0.vpm137.stdout:3821: finishing write tid 7 to vpm1375878-363 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 363 v 435 (ObjNum 2345 snap 0 seq_num 2345) dirty exists 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 1 to vpm1375878-472 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 2 to vpm1375878-472 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 3 to vpm1375878-472 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 4 to vpm1375878-472 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 6 to vpm1375878-472 2014-08-08T10:51:43.725 INFO:tasks.rados.rados.0.vpm137.stdout:3828: finishing write tid 1 to vpm1375878-74 2014-08-08T10:51:43.726 INFO:tasks.rados.rados.0.vpm137.stdout:3828: finishing write tid 3 to vpm1375878-74 2014-08-08T10:51:43.726 INFO:tasks.rados.rados.0.vpm137.stdout:3828: finishing write tid 4 to vpm1375878-74 2014-08-08T10:51:43.726 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 74 v 444 (ObjNum 2349 snap 0 seq_num 2349) dirty exists 2014-08-08T10:51:43.726 INFO:tasks.rados.rados.0.vpm137.stdout:3822: expect (ObjNum 1848 snap 0 seq_num 1848) 2014-08-08T10:51:46.261 INFO:tasks.rados.rados.0.vpm137.stdout:3817: done (11 left) 2014-08-08T10:51:46.261 INFO:tasks.rados.rados.0.vpm137.stdout:3819: done (10 left) 2014-08-08T10:51:46.262 INFO:tasks.rados.rados.0.vpm137.stdout:3821: done (9 left) 2014-08-08T10:51:46.263 INFO:tasks.rados.rados.0.vpm137.stdout:3822: done (8 left) 2014-08-08T10:51:46.264 INFO:tasks.rados.rados.0.vpm137.stdout:3825: done (7 left) 2014-08-08T10:51:46.265 INFO:tasks.rados.rados.0.vpm137.stdout:3828: done (6 left) 2014-08-08T10:51:46.265 INFO:tasks.rados.rados.0.vpm137.stdout:3835: write oid 436 current snap is 0 2014-08-08T10:51:46.265 INFO:tasks.rados.rados.0.vpm137.stdout:3835: seq_num 2356 ranges {497456=420984,1384277=665668,2528486=491016,3588004=109452} 2014-08-08T10:51:46.282 INFO:tasks.rados.rados.0.vpm137.stdout:3835: writing vpm1375878-436 from 497456 to 918440 tid 1 2014-08-08T10:51:46.336 INFO:tasks.rados.rados.0.vpm137.stdout:3835: writing vpm1375878-436 from 1384277 to 2049945 tid 2 2014-08-08T10:51:46.361 INFO:tasks.rados.rados.0.vpm137.stdout:3835: writing vpm1375878-436 from 2528486 to 3019502 tid 3 2014-08-08T10:51:46.370 INFO:tasks.rados.rados.0.vpm137.stdout:3835: writing vpm1375878-436 from 3588004 to 3697456 tid 4 2014-08-08T10:51:46.370 INFO:tasks.rados.rados.0.vpm137.stdout:3823: finishing write tid 7 to vpm1375878-472 2014-08-08T10:51:46.371 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 472 v 440 (ObjNum 2346 snap 0 seq_num 2346) dirty exists 2014-08-08T10:51:46.371 INFO:tasks.rados.rados.0.vpm137.stdout:3830: finishing write tid 1 to vpm1375878-201 2014-08-08T10:51:46.371 INFO:tasks.rados.rados.0.vpm137.stdout:3830: finishing write tid 2 to vpm1375878-201 2014-08-08T10:51:46.371 INFO:tasks.rados.rados.0.vpm137.stdout:3830: finishing write tid 3 to vpm1375878-201 2014-08-08T10:51:46.372 INFO:tasks.rados.rados.0.vpm137.stdout:3830: finishing write tid 5 to vpm1375878-201 2014-08-08T10:51:46.372 INFO:tasks.rados.rados.0.vpm137.stdout:3830: finishing write tid 6 to vpm1375878-201 2014-08-08T10:51:46.372 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 201 v 448 (ObjNum 2351 snap 0 seq_num 2351) dirty exists 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:3831: finishing write tid 1 to vpm1375878-333 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:3831: finishing write tid 2 to vpm1375878-333 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:3831: finishing write tid 4 to vpm1375878-333 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:3831: finishing write tid 5 to vpm1375878-333 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 333 v 469 (ObjNum 2352 snap 0 seq_num 2352) dirty exists 2014-08-08T10:51:46.373 INFO:tasks.rados.rados.0.vpm137.stdout:3832: finishing write tid 1 to vpm1375878-355 2014-08-08T10:51:46.374 INFO:tasks.rados.rados.0.vpm137.stdout:3832: finishing write tid 2 to vpm1375878-355 2014-08-08T10:51:46.374 INFO:tasks.rados.rados.0.vpm137.stdout:3832: finishing write tid 3 to vpm1375878-355 2014-08-08T10:51:46.374 INFO:tasks.rados.rados.0.vpm137.stdout:3832: finishing write tid 5 to vpm1375878-355 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3832: finishing write tid 6 to vpm1375878-355 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 355 v 361 (ObjNum 2353 snap 0 seq_num 2353) dirty exists 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3833: finishing write tid 1 to vpm1375878-413 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3833: finishing write tid 3 to vpm1375878-413 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3834: finishing write tid 1 to vpm1375878-36 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3834: finishing write tid 3 to vpm1375878-36 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:3834: finishing write tid 4 to vpm1375878-36 2014-08-08T10:51:46.375 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 36 v 437 (ObjNum 2355 snap 0 seq_num 2355) dirty exists 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3833: finishing write tid 4 to vpm1375878-413 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 413 v 513 (ObjNum 2354 snap 0 seq_num 2354) dirty exists 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 1 to vpm1375878-436 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3823: done (6 left) 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3830: done (5 left) 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3831: done (4 left) 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3832: done (3 left) 2014-08-08T10:51:46.376 INFO:tasks.rados.rados.0.vpm137.stdout:3833: done (2 left) 2014-08-08T10:51:46.377 INFO:tasks.rados.rados.0.vpm137.stdout:3834: done (1 left) 2014-08-08T10:51:46.377 INFO:tasks.rados.rados.0.vpm137.stdout:3836: read oid 123 snap -1 2014-08-08T10:51:46.377 INFO:tasks.rados.rados.0.vpm137.stdout:3837: write oid 430 current snap is 0 2014-08-08T10:51:46.378 INFO:tasks.rados.rados.0.vpm137.stdout:3837: seq_num 2357 ranges {689838=792789,1489837=1} 2014-08-08T10:51:46.408 INFO:tasks.rados.rados.0.vpm137.stdout:3837: writing vpm1375878-430 from 689838 to 1482627 tid 1 2014-08-08T10:51:46.410 INFO:tasks.rados.rados.0.vpm137.stdout:3837: writing vpm1375878-430 from 1489837 to 1489838 tid 2 2014-08-08T10:51:46.411 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 2 to vpm1375878-436 2014-08-08T10:51:46.411 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 3 to vpm1375878-436 2014-08-08T10:51:46.411 INFO:tasks.rados.rados.0.vpm137.stdout:3838: write oid 237 current snap is 0 2014-08-08T10:51:46.412 INFO:tasks.rados.rados.0.vpm137.stdout:3838: seq_num 2358 ranges {502444=799665,1702443=1} 2014-08-08T10:51:46.447 INFO:tasks.rados.rados.0.vpm137.stdout:3838: writing vpm1375878-237 from 502444 to 1302109 tid 1 2014-08-08T10:51:46.453 INFO:tasks.rados.rados.0.vpm137.stdout:3838: writing vpm1375878-237 from 1702443 to 1702444 tid 2 2014-08-08T10:51:46.453 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 4 to vpm1375878-436 2014-08-08T10:51:46.454 INFO:tasks.rados.rados.0.vpm137.stdout:3836: expect (ObjNum 1890 snap 0 seq_num 1890) 2014-08-08T10:51:46.518 INFO:tasks.rados.rados.0.vpm137.stdout:3836: done (3 left) 2014-08-08T10:51:46.518 INFO:tasks.rados.rados.0.vpm137.stdout:3839: read oid 71 snap -1 2014-08-08T10:51:46.518 INFO:tasks.rados.rados.0.vpm137.stdout:3840: write oid 4 current snap is 0 2014-08-08T10:51:46.518 INFO:tasks.rados.rados.0.vpm137.stdout:3840: seq_num 2359 ranges {597674=772493,1397673=1} 2014-08-08T10:51:46.551 INFO:tasks.rados.rados.0.vpm137.stdout:3840: writing vpm1375878-4 from 597674 to 1370167 tid 1 2014-08-08T10:51:46.552 INFO:tasks.rados.rados.0.vpm137.stdout:3840: writing vpm1375878-4 from 1397673 to 1397674 tid 2 2014-08-08T10:51:46.552 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 6 to vpm1375878-436 2014-08-08T10:51:46.552 INFO:tasks.rados.rados.0.vpm137.stdout:3835: finishing write tid 7 to vpm1375878-436 2014-08-08T10:51:46.553 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 436 v 630 (ObjNum 2356 snap 0 seq_num 2356) dirty exists 2014-08-08T10:51:46.553 INFO:tasks.rados.rados.0.vpm137.stdout:3837: finishing write tid 1 to vpm1375878-430 2014-08-08T10:51:46.553 INFO:tasks.rados.rados.0.vpm137.stdout:3837: finishing write tid 2 to vpm1375878-430 2014-08-08T10:51:46.553 INFO:tasks.rados.rados.0.vpm137.stdout:3837: finishing write tid 4 to vpm1375878-430 2014-08-08T10:51:46.553 INFO:tasks.rados.rados.0.vpm137.stdout:3837: finishing write tid 5 to vpm1375878-430 2014-08-08T10:51:46.554 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 430 v 451 (ObjNum 2357 snap 0 seq_num 2357) dirty exists 2014-08-08T10:51:46.554 INFO:tasks.rados.rados.0.vpm137.stdout:3835: done (4 left) 2014-08-08T10:51:46.554 INFO:tasks.rados.rados.0.vpm137.stdout:3837: done (3 left) 2014-08-08T10:51:46.554 INFO:tasks.rados.rados.0.vpm137.stdout:3841: write oid 415 current snap is 0 2014-08-08T10:51:46.555 INFO:tasks.rados.rados.0.vpm137.stdout:3841: seq_num 2360 ranges {791079=495722,2017190=732945,3191078=1} 2014-08-08T10:51:46.556 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:46.933 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:46.989 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:47.217 INFO:tasks.rados.rados.0.vpm137.stdout:3841: writing vpm1375878-415 from 791079 to 1286801 tid 1 2014-08-08T10:51:47.258 INFO:tasks.rados.rados.0.vpm137.stdout:3841: writing vpm1375878-415 from 2017190 to 2750135 tid 2 2014-08-08T10:51:47.264 INFO:tasks.rados.rados.0.vpm137.stdout:3841: writing vpm1375878-415 from 3191078 to 3191079 tid 3 2014-08-08T10:51:47.266 INFO:tasks.rados.rados.0.vpm137.stdout:3838: finishing write tid 1 to vpm1375878-237 2014-08-08T10:51:47.266 INFO:tasks.rados.rados.0.vpm137.stdout:3838: finishing write tid 2 to vpm1375878-237 2014-08-08T10:51:47.266 INFO:tasks.rados.rados.0.vpm137.stdout:3838: finishing write tid 4 to vpm1375878-237 2014-08-08T10:51:47.266 INFO:tasks.rados.rados.0.vpm137.stdout:3840: finishing write tid 1 to vpm1375878-4 2014-08-08T10:51:47.267 INFO:tasks.rados.rados.0.vpm137.stdout:3840: finishing write tid 2 to vpm1375878-4 2014-08-08T10:51:47.267 INFO:tasks.rados.rados.0.vpm137.stdout:3840: finishing write tid 4 to vpm1375878-4 2014-08-08T10:51:47.268 INFO:tasks.rados.rados.0.vpm137.stdout:3840: finishing write tid 5 to vpm1375878-4 2014-08-08T10:51:47.268 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 4 v 443 (ObjNum 2359 snap 0 seq_num 2359) dirty exists 2014-08-08T10:51:47.268 INFO:tasks.rados.rados.0.vpm137.stdout:3838: finishing write tid 5 to vpm1375878-237 2014-08-08T10:51:47.268 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 237 v 633 (ObjNum 2358 snap 0 seq_num 2358) dirty exists 2014-08-08T10:51:47.268 INFO:tasks.rados.rados.0.vpm137.stdout:3839: expect (ObjNum 70 snap 0 seq_num 70) 2014-08-08T10:51:47.274 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 848103255, u'bytes_used': 1701306368, u'op_per_sec': 12, u'version': 1693, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1896650, u'write_bytes_sec': 1220452, u'bytes_avail': 427585708032}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:47.275 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:47.768 INFO:tasks.rados.rados.0.vpm137.stdout:3838: done (3 left) 2014-08-08T10:51:47.769 INFO:tasks.rados.rados.0.vpm137.stdout:3839: done (2 left) 2014-08-08T10:51:47.769 INFO:tasks.rados.rados.0.vpm137.stdout:3840: done (1 left) 2014-08-08T10:51:47.769 INFO:tasks.rados.rados.0.vpm137.stdout:3842: write oid 190 current snap is 0 2014-08-08T10:51:47.769 INFO:tasks.rados.rados.0.vpm137.stdout:3842: seq_num 2361 ranges {487333=618950,1731322=403140,2562966=324367} 2014-08-08T10:51:47.804 INFO:tasks.rados.rados.0.vpm137.stdout:3842: writing vpm1375878-190 from 487333 to 1106283 tid 1 2014-08-08T10:51:47.845 INFO:tasks.rados.rados.0.vpm137.stdout:3842: writing vpm1375878-190 from 1731322 to 2134462 tid 2 2014-08-08T10:51:47.860 INFO:tasks.rados.rados.0.vpm137.stdout:3842: writing vpm1375878-190 from 2562966 to 2887333 tid 3 2014-08-08T10:51:47.861 INFO:tasks.rados.rados.0.vpm137.stdout:3841: finishing write tid 1 to vpm1375878-415 2014-08-08T10:51:47.862 INFO:tasks.rados.rados.0.vpm137.stdout:3841: finishing write tid 2 to vpm1375878-415 2014-08-08T10:51:47.862 INFO:tasks.rados.rados.0.vpm137.stdout:3841: finishing write tid 3 to vpm1375878-415 2014-08-08T10:51:47.862 INFO:tasks.rados.rados.0.vpm137.stdout:3841: finishing write tid 5 to vpm1375878-415 2014-08-08T10:51:47.862 INFO:tasks.rados.rados.0.vpm137.stdout:3841: finishing write tid 6 to vpm1375878-415 2014-08-08T10:51:47.863 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 415 v 441 (ObjNum 2360 snap 0 seq_num 2360) dirty exists 2014-08-08T10:51:47.863 INFO:tasks.rados.rados.0.vpm137.stdout:3842: finishing write tid 1 to vpm1375878-190 2014-08-08T10:51:47.864 INFO:tasks.rados.rados.0.vpm137.stdout:3841: done (1 left) 2014-08-08T10:51:47.864 INFO:tasks.rados.rados.0.vpm137.stdout:3843: write oid 257 current snap is 0 2014-08-08T10:51:47.864 INFO:tasks.rados.rados.0.vpm137.stdout:3843: seq_num 2362 ranges {582563=724002,2094030=488533} 2014-08-08T10:51:47.891 INFO:tasks.rados.rados.0.vpm137.stdout:3843: writing vpm1375878-257 from 582563 to 1306565 tid 1 2014-08-08T10:51:47.917 INFO:tasks.rados.rados.0.vpm137.stdout:3843: writing vpm1375878-257 from 2094030 to 2582563 tid 2 2014-08-08T10:51:47.918 INFO:tasks.rados.rados.0.vpm137.stdout:3844: delete oid 272 current snap is 0 2014-08-08T10:51:47.931 INFO:tasks.rados.rados.0.vpm137.stdout:3842: finishing write tid 2 to vpm1375878-190 2014-08-08T10:51:47.939 INFO:tasks.rados.rados.0.vpm137.stdout:3842: finishing write tid 3 to vpm1375878-190 2014-08-08T10:51:47.939 INFO:tasks.rados.rados.0.vpm137.stdout:3842: finishing write tid 5 to vpm1375878-190 2014-08-08T10:51:47.950 INFO:tasks.rados.rados.0.vpm137.stdout:3843: finishing write tid 1 to vpm1375878-257 2014-08-08T10:51:47.964 INFO:tasks.rados.rados.0.vpm137.stdout:3842: finishing write tid 6 to vpm1375878-190 2014-08-08T10:51:47.964 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 190 v 455 (ObjNum 2361 snap 0 seq_num 2361) dirty exists 2014-08-08T10:51:48.001 INFO:tasks.rados.rados.0.vpm137.stdout:3843: finishing write tid 2 to vpm1375878-257 2014-08-08T10:51:48.026 INFO:tasks.rados.rados.0.vpm137.stdout:3843: finishing write tid 4 to vpm1375878-257 2014-08-08T10:51:48.027 INFO:tasks.rados.rados.0.vpm137.stdout:3842: done (2 left) 2014-08-08T10:51:48.027 INFO:tasks.rados.rados.0.vpm137.stdout:3844: done (1 left) 2014-08-08T10:51:48.027 INFO:tasks.rados.rados.0.vpm137.stdout:3845: read oid 103 snap -1 2014-08-08T10:51:48.028 INFO:tasks.rados.rados.0.vpm137.stdout:3846: read oid 109 snap -1 2014-08-08T10:51:48.028 INFO:tasks.rados.rados.0.vpm137.stdout:3847: read oid 1 snap -1 2014-08-08T10:51:48.029 INFO:tasks.rados.rados.0.vpm137.stdout:3848: read oid 138 snap -1 2014-08-08T10:51:48.029 INFO:tasks.rados.rados.0.vpm137.stdout:3849: read oid 292 snap -1 2014-08-08T10:51:48.029 INFO:tasks.rados.rados.0.vpm137.stdout:3850: read oid 150 snap -1 2014-08-08T10:51:48.030 INFO:tasks.rados.rados.0.vpm137.stdout:3851: write oid 442 current snap is 0 2014-08-08T10:51:48.030 INFO:tasks.rados.rados.0.vpm137.stdout:3851: seq_num 2364 ranges {586525=1} 2014-08-08T10:51:48.034 INFO:tasks.rados.rados.0.vpm137.stdout:3851: writing vpm1375878-442 from 586525 to 586526 tid 1 2014-08-08T10:51:48.035 INFO:tasks.rados.rados.0.vpm137.stdout:3852: read oid 235 snap -1 2014-08-08T10:51:48.036 INFO:tasks.rados.rados.0.vpm137.stdout:3853: write oid 51 current snap is 0 2014-08-08T10:51:48.036 INFO:tasks.rados.rados.0.vpm137.stdout:3853: seq_num 2365 ranges {282779=1} 2014-08-08T10:51:48.038 INFO:tasks.rados.rados.0.vpm137.stdout:3853: writing vpm1375878-51 from 282779 to 282780 tid 1 2014-08-08T10:51:48.038 INFO:tasks.rados.rados.0.vpm137.stdout:3854: read oid 341 snap -1 2014-08-08T10:51:48.039 INFO:tasks.rados.rados.0.vpm137.stdout:3855: read oid 158 snap -1 2014-08-08T10:51:48.039 INFO:tasks.rados.rados.0.vpm137.stdout:3856: read oid 172 snap -1 2014-08-08T10:51:48.040 INFO:tasks.rados.rados.0.vpm137.stdout:3857: read oid 282 snap -1 2014-08-08T10:51:48.040 INFO:tasks.rados.rados.0.vpm137.stdout:3858: write oid 295 current snap is 0 2014-08-08T10:51:48.041 INFO:tasks.rados.rados.0.vpm137.stdout:3858: seq_num 2366 ranges {476186=798740,1811615=264571} 2014-08-08T10:51:48.073 INFO:tasks.rados.rados.0.vpm137.stdout:3858: writing vpm1375878-295 from 476186 to 1274926 tid 1 2014-08-08T10:51:48.089 INFO:tasks.rados.rados.0.vpm137.stdout:3858: writing vpm1375878-295 from 1811615 to 2076186 tid 2 2014-08-08T10:51:48.090 INFO:tasks.rados.rados.0.vpm137.stdout:3859: delete oid 56 current snap is 0 2014-08-08T10:51:48.303 INFO:tasks.rados.rados.0.vpm137.stdout:3843: finishing write tid 5 to vpm1375878-257 2014-08-08T10:51:48.304 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 257 v 457 (ObjNum 2362 snap 0 seq_num 2362) dirty exists 2014-08-08T10:51:48.495 INFO:tasks.rados.rados.0.vpm137.stdout:3845: expect (ObjNum 2099 snap 0 seq_num 2099) 2014-08-08T10:51:49.962 INFO:tasks.rados.rados.0.vpm137.stdout:3843: done (15 left) 2014-08-08T10:51:49.962 INFO:tasks.rados.rados.0.vpm137.stdout:3845: done (14 left) 2014-08-08T10:51:49.962 INFO:tasks.rados.rados.0.vpm137.stdout:3859: done (13 left) 2014-08-08T10:51:49.963 INFO:tasks.rados.rados.0.vpm137.stdout:3860: read oid 53 snap -1 2014-08-08T10:51:49.963 INFO:tasks.rados.rados.0.vpm137.stdout:3861: read oid 128 snap -1 2014-08-08T10:51:49.963 INFO:tasks.rados.rados.0.vpm137.stdout:3862: write oid 97 current snap is 0 2014-08-08T10:51:49.963 INFO:tasks.rados.rados.0.vpm137.stdout:3862: seq_num 2368 ranges {632597=493772,1719110=432426,2717270=315327} 2014-08-08T10:51:49.984 INFO:tasks.rados.rados.0.vpm137.stdout:3862: writing vpm1375878-97 from 632597 to 1126369 tid 1 2014-08-08T10:51:50.006 INFO:tasks.rados.rados.0.vpm137.stdout:3862: writing vpm1375878-97 from 1719110 to 2151536 tid 2 2014-08-08T10:51:50.022 INFO:tasks.rados.rados.0.vpm137.stdout:3862: writing vpm1375878-97 from 2717270 to 3032597 tid 3 2014-08-08T10:51:50.023 INFO:tasks.rados.rados.0.vpm137.stdout:3849: expect (ObjNum 2294 snap 0 seq_num 2294) 2014-08-08T10:51:50.466 INFO:tasks.rados.rados.0.vpm137.stdout:3847: done (15 left) 2014-08-08T10:51:50.467 INFO:tasks.rados.rados.0.vpm137.stdout:3849: done (14 left) 2014-08-08T10:51:50.467 INFO:tasks.rados.rados.0.vpm137.stdout:3863: write oid 279 current snap is 0 2014-08-08T10:51:50.468 INFO:tasks.rados.rados.0.vpm137.stdout:3863: seq_num 2369 ranges {727827=618025,1984667=502621,2727826=1} 2014-08-08T10:51:50.499 INFO:tasks.rados.rados.0.vpm137.stdout:3863: writing vpm1375878-279 from 727827 to 1345852 tid 1 2014-08-08T10:51:50.536 INFO:tasks.rados.rados.0.vpm137.stdout:3863: writing vpm1375878-279 from 1984667 to 2487288 tid 2 2014-08-08T10:51:50.540 INFO:tasks.rados.rados.0.vpm137.stdout:3863: writing vpm1375878-279 from 2727826 to 2727827 tid 3 2014-08-08T10:51:50.541 INFO:tasks.rados.rados.0.vpm137.stdout:3846: expect (ObjNum 1424 snap 0 seq_num 1424) 2014-08-08T10:51:50.557 INFO:tasks.rados.rados.0.vpm137.stdout:3846: done (14 left) 2014-08-08T10:51:50.557 INFO:tasks.rados.rados.0.vpm137.stdout:3864: write oid 492 current snap is 0 2014-08-08T10:51:50.558 INFO:tasks.rados.rados.0.vpm137.stdout:3864: seq_num 2370 ranges {672657=473477,1547376=572816,2851946=620711} 2014-08-08T10:51:50.590 INFO:tasks.rados.rados.0.vpm137.stdout:3864: writing vpm1375878-492 from 672657 to 1146134 tid 1 2014-08-08T10:51:50.627 INFO:tasks.rados.rados.0.vpm137.stdout:3864: writing vpm1375878-492 from 1547376 to 2120192 tid 2 2014-08-08T10:51:50.668 INFO:tasks.rados.rados.0.vpm137.stdout:3864: writing vpm1375878-492 from 2851946 to 3472657 tid 3 2014-08-08T10:51:50.670 INFO:tasks.rados.rados.0.vpm137.stdout:3852: expect (ObjNum 2327 snap 0 seq_num 2327) 2014-08-08T10:51:50.698 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:50.699 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:51.097 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:52.734 INFO:tasks.rados.rados.0.vpm137.stdout:3852: done (14 left) 2014-08-08T10:51:52.734 INFO:tasks.rados.rados.0.vpm137.stdout:3865: read oid 196 snap -1 2014-08-08T10:51:52.734 INFO:tasks.rados.rados.0.vpm137.stdout:3866: read oid 37 snap -1 2014-08-08T10:51:52.734 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:52.735 INFO:tasks.rados.rados.0.vpm137.stdout:3854: expect (ObjNum 340 snap 0 seq_num 340) 2014-08-08T10:51:53.757 INFO:tasks.rados.rados.0.vpm137.stdout:3854: done (15 left) 2014-08-08T10:51:53.758 INFO:tasks.rados.rados.0.vpm137.stdout:3867: read oid 475 snap -1 2014-08-08T10:51:53.761 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:53.761 INFO:tasks.rados.rados.0.vpm137.stdout:3855: done (15 left) 2014-08-08T10:51:53.761 INFO:tasks.rados.rados.0.vpm137.stdout:3868: read oid 396 snap -1 2014-08-08T10:51:53.761 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:53.762 INFO:tasks.rados.rados.0.vpm137.stdout:3851: finishing write tid 1 to vpm1375878-442 2014-08-08T10:51:53.762 INFO:tasks.rados.rados.0.vpm137.stdout:3851: finishing write tid 3 to vpm1375878-442 2014-08-08T10:51:53.762 INFO:tasks.rados.rados.0.vpm137.stdout:3853: finishing write tid 1 to vpm1375878-51 2014-08-08T10:51:53.762 INFO:tasks.rados.rados.0.vpm137.stdout:3853: finishing write tid 3 to vpm1375878-51 2014-08-08T10:51:53.763 INFO:tasks.rados.rados.0.vpm137.stdout:3858: finishing write tid 1 to vpm1375878-295 2014-08-08T10:51:53.763 INFO:tasks.rados.rados.0.vpm137.stdout:3858: finishing write tid 2 to vpm1375878-295 2014-08-08T10:51:53.763 INFO:tasks.rados.rados.0.vpm137.stdout:3858: finishing write tid 4 to vpm1375878-295 2014-08-08T10:51:53.763 INFO:tasks.rados.rados.0.vpm137.stdout:3851: finishing write tid 4 to vpm1375878-442 2014-08-08T10:51:53.764 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 442 v 317 (ObjNum 2364 snap 0 seq_num 2364) dirty exists 2014-08-08T10:51:53.764 INFO:tasks.rados.rados.0.vpm137.stdout:3851: done (15 left) 2014-08-08T10:51:53.764 INFO:tasks.rados.rados.0.vpm137.stdout:3869: read oid 482 snap -1 2014-08-08T10:51:53.764 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:53.765 INFO:tasks.rados.rados.0.vpm137.stdout:3853: finishing write tid 4 to vpm1375878-51 2014-08-08T10:51:53.765 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 51 v 319 (ObjNum 2365 snap 0 seq_num 2365) dirty exists 2014-08-08T10:51:53.765 INFO:tasks.rados.rados.0.vpm137.stdout:3853: done (15 left) 2014-08-08T10:51:53.765 INFO:tasks.rados.rados.0.vpm137.stdout:3870: delete oid 147 current snap is 0 2014-08-08T10:51:53.765 INFO:tasks.rados.rados.0.vpm137.stdout:3857: expect (ObjNum 1157 snap 0 seq_num 1157) 2014-08-08T10:51:54.128 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:54.332 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout:3857: done (15 left) 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout:3870: done (14 left) 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout:3871: read oid 471 snap -1 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout:3872: read oid 451 snap -1 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:54.338 INFO:tasks.rados.rados.0.vpm137.stdout:3850: expect (ObjNum 2057 snap 0 seq_num 2057) 2014-08-08T10:51:54.397 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:54.618 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016391', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 850012233, u'bytes_used': 1700134912, u'op_per_sec': 15, u'version': 1694, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2678962, u'write_bytes_sec': 1158938, u'bytes_avail': 427586879488}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:54.619 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:54.829 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:54.829 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:55.030 INFO:tasks.rados.rados.0.vpm137.stdout:3850: done (15 left) 2014-08-08T10:51:55.031 INFO:tasks.rados.rados.0.vpm137.stdout:3873: write oid 41 current snap is 0 2014-08-08T10:51:55.031 INFO:tasks.rados.rados.0.vpm137.stdout:3873: seq_num 2372 ranges {428043=1} 2014-08-08T10:51:55.034 INFO:tasks.rados.rados.0.vpm137.stdout:3873: writing vpm1375878-41 from 428043 to 428044 tid 1 2014-08-08T10:51:55.035 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:51:55.035 INFO:tasks.rados.rados.0.vpm137.stdout:3858: finishing write tid 5 to vpm1375878-295 2014-08-08T10:51:55.035 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 295 v 375 (ObjNum 2366 snap 0 seq_num 2366) dirty exists 2014-08-08T10:51:55.036 INFO:tasks.rados.rados.0.vpm137.stdout:3848: expect (ObjNum 137 snap 0 seq_num 137) 2014-08-08T10:51:55.077 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:55.988 INFO:tasks.rados.rados.0.vpm137.stdout:3848: done (15 left) 2014-08-08T10:51:55.989 INFO:tasks.rados.rados.0.vpm137.stdout:3858: done (14 left) 2014-08-08T10:51:55.989 INFO:tasks.rados.rados.0.vpm137.stdout:3874: write oid 144 current snap is 0 2014-08-08T10:51:55.989 INFO:tasks.rados.rados.0.vpm137.stdout:3874: seq_num 2373 ranges {772874=400000} 2014-08-08T10:51:56.007 INFO:tasks.rados.rados.0.vpm137.stdout:3874: writing vpm1375878-144 from 772874 to 1172874 tid 1 2014-08-08T10:51:56.009 INFO:tasks.rados.rados.0.vpm137.stdout:3856: expect (ObjNum 921 snap 0 seq_num 921) 2014-08-08T10:51:57.367 INFO:tasks.rados.rados.0.vpm137.stdout:3856: done (14 left) 2014-08-08T10:51:57.368 INFO:tasks.rados.rados.0.vpm137.stdout:3875: write oid 73 current snap is 0 2014-08-08T10:51:57.368 INFO:tasks.rados.rados.0.vpm137.stdout:3875: seq_num 2374 ranges {717704=797814,1917703=1} 2014-08-08T10:51:57.401 INFO:tasks.rados.rados.0.vpm137.stdout:3875: writing vpm1375878-73 from 717704 to 1515518 tid 1 2014-08-08T10:51:57.407 INFO:tasks.rados.rados.0.vpm137.stdout:3875: writing vpm1375878-73 from 1917703 to 1917704 tid 2 2014-08-08T10:51:57.408 INFO:tasks.rados.rados.0.vpm137.stdout:3861: expect (ObjNum 1987 snap 0 seq_num 1987) 2014-08-08T10:51:57.415 INFO:tasks.rados.rados.0.vpm137.stdout:3861: done (14 left) 2014-08-08T10:51:57.415 INFO:tasks.rados.rados.0.vpm137.stdout:3876: read oid 59 snap -1 2014-08-08T10:51:57.416 INFO:tasks.rados.rados.0.vpm137.stdout:3877: write oid 354 current snap is 0 2014-08-08T10:51:57.416 INFO:tasks.rados.rados.0.vpm137.stdout:3877: seq_num 2375 ranges {412933=521043,1494420=118513} 2014-08-08T10:51:57.438 INFO:tasks.rados.rados.0.vpm137.stdout:3877: writing vpm1375878-354 from 412933 to 933976 tid 1 2014-08-08T10:51:57.454 INFO:tasks.rados.rados.0.vpm137.stdout:3877: writing vpm1375878-354 from 1494420 to 1612933 tid 2 2014-08-08T10:51:57.456 INFO:tasks.rados.rados.0.vpm137.stdout:3862: finishing write tid 1 to vpm1375878-97 2014-08-08T10:51:57.457 INFO:tasks.rados.rados.0.vpm137.stdout:3860: expect (ObjNum 1887 snap 0 seq_num 1887) 2014-08-08T10:51:58.119 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:58.361 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:58.412 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:58.666 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 850156851, u'bytes_used': 1698033664, u'op_per_sec': 17, u'version': 1695, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 3111535, u'write_bytes_sec': 1294571, u'bytes_avail': 427588980736}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:51:58.666 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:51:58.896 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:51:58.896 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:51:58.905 INFO:tasks.rados.rados.0.vpm137.stdout:3860: done (15 left) 2014-08-08T10:51:58.905 INFO:tasks.rados.rados.0.vpm137.stdout:3878: write oid 47 current snap is 0 2014-08-08T10:51:58.905 INFO:tasks.rados.rados.0.vpm137.stdout:3878: seq_num 2376 ranges {474115=493871,1574504=647236,2825251=48864} 2014-08-08T10:51:58.924 INFO:tasks.rados.rados.0.vpm137.stdout:3878: writing vpm1375878-47 from 474115 to 967986 tid 1 2014-08-08T10:51:58.953 INFO:tasks.rados.rados.0.vpm137.stdout:3878: writing vpm1375878-47 from 1574504 to 2221740 tid 2 2014-08-08T10:51:58.960 INFO:tasks.rados.rados.0.vpm137.stdout:3878: writing vpm1375878-47 from 2825251 to 2874115 tid 3 2014-08-08T10:51:58.962 INFO:tasks.rados.rados.0.vpm137.stdout:3862: finishing write tid 2 to vpm1375878-97 2014-08-08T10:51:58.962 INFO:tasks.rados.rados.0.vpm137.stdout:3862: finishing write tid 3 to vpm1375878-97 2014-08-08T10:51:58.962 INFO:tasks.rados.rados.0.vpm137.stdout:3862: finishing write tid 5 to vpm1375878-97 2014-08-08T10:51:58.963 INFO:tasks.rados.rados.0.vpm137.stdout:3862: finishing write tid 6 to vpm1375878-97 2014-08-08T10:51:58.963 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 97 v 637 (ObjNum 2368 snap 0 seq_num 2368) dirty exists 2014-08-08T10:51:58.964 INFO:tasks.rados.rados.0.vpm137.stdout:3863: finishing write tid 1 to vpm1375878-279 2014-08-08T10:51:58.965 INFO:tasks.rados.rados.0.vpm137.stdout:3863: finishing write tid 2 to vpm1375878-279 2014-08-08T10:51:58.965 INFO:tasks.rados.rados.0.vpm137.stdout:3863: finishing write tid 3 to vpm1375878-279 2014-08-08T10:51:58.965 INFO:tasks.rados.rados.0.vpm137.stdout:3863: finishing write tid 5 to vpm1375878-279 2014-08-08T10:51:58.965 INFO:tasks.rados.rados.0.vpm137.stdout:3863: finishing write tid 6 to vpm1375878-279 2014-08-08T10:51:58.965 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 279 v 379 (ObjNum 2369 snap 0 seq_num 2369) dirty exists 2014-08-08T10:51:58.966 INFO:tasks.rados.rados.0.vpm137.stdout:3864: finishing write tid 1 to vpm1375878-492 2014-08-08T10:51:58.966 INFO:tasks.rados.rados.0.vpm137.stdout:3864: finishing write tid 2 to vpm1375878-492 2014-08-08T10:51:58.966 INFO:tasks.rados.rados.0.vpm137.stdout:3864: finishing write tid 3 to vpm1375878-492 2014-08-08T10:51:58.966 INFO:tasks.rados.rados.0.vpm137.stdout:3864: finishing write tid 5 to vpm1375878-492 2014-08-08T10:51:58.966 INFO:tasks.rados.rados.0.vpm137.stdout:3864: finishing write tid 6 to vpm1375878-492 2014-08-08T10:51:58.967 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 492 v 365 (ObjNum 2370 snap 0 seq_num 2370) dirty exists 2014-08-08T10:51:58.967 INFO:tasks.rados.rados.0.vpm137.stdout:3866: expect (ObjNum 2243 snap 0 seq_num 2243) 2014-08-08T10:51:59.156 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:51:59.598 INFO:tasks.rados.rados.0.vpm137.stdout:3862: done (15 left) 2014-08-08T10:51:59.598 INFO:tasks.rados.rados.0.vpm137.stdout:3863: done (14 left) 2014-08-08T10:51:59.598 INFO:tasks.rados.rados.0.vpm137.stdout:3864: done (13 left) 2014-08-08T10:51:59.599 INFO:tasks.rados.rados.0.vpm137.stdout:3866: done (12 left) 2014-08-08T10:51:59.599 INFO:tasks.rados.rados.0.vpm137.stdout:3879: read oid 148 snap -1 2014-08-08T10:51:59.599 INFO:tasks.rados.rados.0.vpm137.stdout:3880: read oid 402 snap -1 2014-08-08T10:51:59.599 INFO:tasks.rados.rados.0.vpm137.stdout:3881: write oid 440 current snap is 0 2014-08-08T10:51:59.599 INFO:tasks.rados.rados.0.vpm137.stdout:3881: seq_num 2377 ranges {417921=617099,1652540=449655,2571180=445393,3617920=1} 2014-08-08T10:51:59.637 INFO:tasks.rados.rados.0.vpm137.stdout:3881: writing vpm1375878-440 from 417921 to 1035020 tid 1 2014-08-08T10:51:59.673 INFO:tasks.rados.rados.0.vpm137.stdout:3881: writing vpm1375878-440 from 1652540 to 2102195 tid 2 2014-08-08T10:51:59.697 INFO:tasks.rados.rados.0.vpm137.stdout:3881: writing vpm1375878-440 from 2571180 to 3016573 tid 3 2014-08-08T10:51:59.703 INFO:tasks.rados.rados.0.vpm137.stdout:3881: writing vpm1375878-440 from 3617920 to 3617921 tid 4 2014-08-08T10:51:59.703 INFO:tasks.rados.rados.0.vpm137.stdout:3865: expect (ObjNum 1802 snap 0 seq_num 1802) 2014-08-08T10:52:00.062 INFO:tasks.rados.rados.0.vpm137.stdout:3865: done (14 left) 2014-08-08T10:52:00.062 INFO:tasks.rados.rados.0.vpm137.stdout:3882: read oid 58 snap -1 2014-08-08T10:52:00.062 INFO:tasks.rados.rados.0.vpm137.stdout:3883: read oid 82 snap -1 2014-08-08T10:52:00.063 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:00.063 INFO:tasks.rados.rados.0.vpm137.stdout:3867: done (15 left) 2014-08-08T10:52:00.063 INFO:tasks.rados.rados.0.vpm137.stdout:3884: read oid 91 snap -1 2014-08-08T10:52:00.064 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:00.064 INFO:tasks.rados.rados.0.vpm137.stdout:3868: done (15 left) 2014-08-08T10:52:00.065 INFO:tasks.rados.rados.0.vpm137.stdout:3885: read oid 182 snap -1 2014-08-08T10:52:00.065 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:00.065 INFO:tasks.rados.rados.0.vpm137.stdout:3869: expect (ObjNum 2332 snap 0 seq_num 2332) 2014-08-08T10:52:00.129 INFO:tasks.rados.rados.0.vpm137.stdout:3869: done (15 left) 2014-08-08T10:52:00.130 INFO:tasks.rados.rados.0.vpm137.stdout:3886: write oid 120 current snap is 0 2014-08-08T10:52:00.130 INFO:tasks.rados.rados.0.vpm137.stdout:3886: seq_num 2378 ranges {514175=472552,1650321=671274,2807478=506697} 2014-08-08T10:52:00.148 INFO:tasks.rados.rados.0.vpm137.stdout:3886: writing vpm1375878-120 from 514175 to 986727 tid 1 2014-08-08T10:52:00.178 INFO:tasks.rados.rados.0.vpm137.stdout:3886: writing vpm1375878-120 from 1650321 to 2321595 tid 2 2014-08-08T10:52:00.202 INFO:tasks.rados.rados.0.vpm137.stdout:3886: writing vpm1375878-120 from 2807478 to 3314175 tid 3 2014-08-08T10:52:00.204 INFO:tasks.rados.rados.0.vpm137.stdout:3872: expect (ObjNum 1751 snap 0 seq_num 1751) 2014-08-08T10:52:00.332 INFO:tasks.rados.rados.0.vpm137.stdout:3871: done (15 left) 2014-08-08T10:52:00.332 INFO:tasks.rados.rados.0.vpm137.stdout:3872: done (14 left) 2014-08-08T10:52:00.333 INFO:tasks.rados.rados.0.vpm137.stdout:3887: read oid 407 snap -1 2014-08-08T10:52:00.333 INFO:tasks.rados.rados.0.vpm137.stdout:3888: write oid 396 current snap is 0 2014-08-08T10:52:00.333 INFO:tasks.rados.rados.0.vpm137.stdout:3888: seq_num 2379 ranges {610428=595780,1764452=492893,3010427=1} 2014-08-08T10:52:00.368 INFO:tasks.rados.rados.0.vpm137.stdout:3888: writing vpm1375878-396 from 610428 to 1206208 tid 1 2014-08-08T10:52:00.399 INFO:tasks.rados.rados.0.vpm137.stdout:3888: writing vpm1375878-396 from 1764452 to 2257345 tid 2 2014-08-08T10:52:00.410 INFO:tasks.rados.rados.0.vpm137.stdout:3888: writing vpm1375878-396 from 3010427 to 3010428 tid 3 2014-08-08T10:52:00.411 INFO:tasks.rados.rados.0.vpm137.stdout:3873: finishing write tid 1 to vpm1375878-41 2014-08-08T10:52:00.412 INFO:tasks.rados.rados.0.vpm137.stdout:3873: finishing write tid 3 to vpm1375878-41 2014-08-08T10:52:00.412 INFO:tasks.rados.rados.0.vpm137.stdout:3873: finishing write tid 4 to vpm1375878-41 2014-08-08T10:52:00.413 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 41 v 409 (ObjNum 2372 snap 0 seq_num 2372) dirty exists 2014-08-08T10:52:00.413 INFO:tasks.rados.rados.0.vpm137.stdout:3874: finishing write tid 1 to vpm1375878-144 2014-08-08T10:52:00.413 INFO:tasks.rados.rados.0.vpm137.stdout:3874: finishing write tid 3 to vpm1375878-144 2014-08-08T10:52:00.414 INFO:tasks.rados.rados.0.vpm137.stdout:3874: finishing write tid 4 to vpm1375878-144 2014-08-08T10:52:00.414 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 144 v 428 (ObjNum 2373 snap 0 seq_num 2373) dirty exists 2014-08-08T10:52:00.414 INFO:tasks.rados.rados.0.vpm137.stdout:3876: expect (ObjNum 1411 snap 0 seq_num 1411) 2014-08-08T10:52:01.641 INFO:tasks.rados.rados.0.vpm137.stdout:3873: done (15 left) 2014-08-08T10:52:01.641 INFO:tasks.rados.rados.0.vpm137.stdout:3874: done (14 left) 2014-08-08T10:52:01.641 INFO:tasks.rados.rados.0.vpm137.stdout:3876: done (13 left) 2014-08-08T10:52:01.642 INFO:tasks.rados.rados.0.vpm137.stdout:3889: read oid 192 snap -1 2014-08-08T10:52:01.643 INFO:tasks.rados.rados.0.vpm137.stdout:3890: write oid 408 current snap is 0 2014-08-08T10:52:01.643 INFO:tasks.rados.rados.0.vpm137.stdout:3890: seq_num 2380 ranges {402810=400000} 2014-08-08T10:52:01.657 INFO:tasks.rados.rados.0.vpm137.stdout:3890: writing vpm1375878-408 from 402810 to 802810 tid 1 2014-08-08T10:52:01.658 INFO:tasks.rados.rados.0.vpm137.stdout:3875: finishing write tid 1 to vpm1375878-73 2014-08-08T10:52:01.659 INFO:tasks.rados.rados.0.vpm137.stdout:3875: finishing write tid 2 to vpm1375878-73 2014-08-08T10:52:01.659 INFO:tasks.rados.rados.0.vpm137.stdout:3875: finishing write tid 4 to vpm1375878-73 2014-08-08T10:52:01.659 INFO:tasks.rados.rados.0.vpm137.stdout:3877: finishing write tid 1 to vpm1375878-354 2014-08-08T10:52:01.660 INFO:tasks.rados.rados.0.vpm137.stdout:3877: finishing write tid 2 to vpm1375878-354 2014-08-08T10:52:01.660 INFO:tasks.rados.rados.0.vpm137.stdout:3877: finishing write tid 4 to vpm1375878-354 2014-08-08T10:52:01.661 INFO:tasks.rados.rados.0.vpm137.stdout:3875: finishing write tid 5 to vpm1375878-73 2014-08-08T10:52:01.661 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 73 v 446 (ObjNum 2374 snap 0 seq_num 2374) dirty exists 2014-08-08T10:52:01.661 INFO:tasks.rados.rados.0.vpm137.stdout:3877: finishing write tid 5 to vpm1375878-354 2014-08-08T10:52:01.662 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 354 v 472 (ObjNum 2375 snap 0 seq_num 2375) dirty exists 2014-08-08T10:52:01.662 INFO:tasks.rados.rados.0.vpm137.stdout:3878: finishing write tid 1 to vpm1375878-47 2014-08-08T10:52:01.664 INFO:tasks.rados.rados.0.vpm137.stdout:3879: expect (ObjNum 2305 snap 0 seq_num 2305) 2014-08-08T10:52:02.184 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:03.690 INFO:tasks.rados.rados.0.vpm137.stdout:3875: done (14 left) 2014-08-08T10:52:03.690 INFO:tasks.rados.rados.0.vpm137.stdout:3877: done (13 left) 2014-08-08T10:52:03.690 INFO:tasks.rados.rados.0.vpm137.stdout:3879: done (12 left) 2014-08-08T10:52:03.690 INFO:tasks.rados.rados.0.vpm137.stdout:3891: delete oid 64 current snap is 0 2014-08-08T10:52:03.691 INFO:tasks.rados.rados.0.vpm137.stdout:3880: expect (ObjNum 1461 snap 0 seq_num 1461) 2014-08-08T10:52:04.563 INFO:tasks.rados.rados.0.vpm137.stdout:3880: done (12 left) 2014-08-08T10:52:04.563 INFO:tasks.rados.rados.0.vpm137.stdout:3891: done (11 left) 2014-08-08T10:52:04.563 INFO:tasks.rados.rados.0.vpm137.stdout:3892: write oid 387 current snap is 0 2014-08-08T10:52:04.563 INFO:tasks.rados.rados.0.vpm137.stdout:3892: seq_num 2382 ranges {709621=1} 2014-08-08T10:52:04.568 INFO:tasks.rados.rados.0.vpm137.stdout:3892: writing vpm1375878-387 from 709621 to 709622 tid 1 2014-08-08T10:52:04.569 INFO:tasks.rados.rados.0.vpm137.stdout:3878: finishing write tid 2 to vpm1375878-47 2014-08-08T10:52:04.569 INFO:tasks.rados.rados.0.vpm137.stdout:3878: finishing write tid 3 to vpm1375878-47 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:3878: finishing write tid 5 to vpm1375878-47 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:3878: finishing write tid 6 to vpm1375878-47 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 47 v 371 (ObjNum 2376 snap 0 seq_num 2376) dirty exists 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 1 to vpm1375878-440 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 2 to vpm1375878-440 2014-08-08T10:52:04.570 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 3 to vpm1375878-440 2014-08-08T10:52:04.571 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 4 to vpm1375878-440 2014-08-08T10:52:04.571 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 6 to vpm1375878-440 2014-08-08T10:52:04.571 INFO:tasks.rados.rados.0.vpm137.stdout:3881: finishing write tid 7 to vpm1375878-440 2014-08-08T10:52:04.571 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 440 v 370 (ObjNum 2377 snap 0 seq_num 2377) dirty exists 2014-08-08T10:52:04.571 INFO:tasks.rados.rados.0.vpm137.stdout:3884: expect (ObjNum 1933 snap 0 seq_num 1933) 2014-08-08T10:52:05.354 INFO:tasks.rados.rados.0.vpm137.stdout:3878: done (11 left) 2014-08-08T10:52:05.354 INFO:tasks.rados.rados.0.vpm137.stdout:3881: done (10 left) 2014-08-08T10:52:05.354 INFO:tasks.rados.rados.0.vpm137.stdout:3884: done (9 left) 2014-08-08T10:52:05.354 INFO:tasks.rados.rados.0.vpm137.stdout:3893: read oid 237 snap -1 2014-08-08T10:52:05.355 INFO:tasks.rados.rados.0.vpm137.stdout:3894: read oid 427 snap -1 2014-08-08T10:52:05.355 INFO:tasks.rados.rados.0.vpm137.stdout:3895: write oid 232 current snap is 0 2014-08-08T10:52:05.355 INFO:tasks.rados.rados.0.vpm137.stdout:3895: seq_num 2383 ranges {503027=520118,1694517=525098,2503026=1} 2014-08-08T10:52:05.382 INFO:tasks.rados.rados.0.vpm137.stdout:3895: writing vpm1375878-232 from 503027 to 1023145 tid 1 2014-08-08T10:52:05.415 INFO:tasks.rados.rados.0.vpm137.stdout:3895: writing vpm1375878-232 from 1694517 to 2219615 tid 2 2014-08-08T10:52:05.418 INFO:tasks.rados.rados.0.vpm137.stdout:3895: writing vpm1375878-232 from 2503026 to 2503027 tid 3 2014-08-08T10:52:05.419 INFO:tasks.rados.rados.0.vpm137.stdout:3887: expect (ObjNum 1594 snap 0 seq_num 1594) 2014-08-08T10:52:05.463 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:05.524 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:05.769 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 857654826, u'bytes_used': 1701285888, u'op_per_sec': 10, u'version': 1697, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2336972, u'write_bytes_sec': 872733, u'bytes_avail': 427585728512}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:05.770 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:05.989 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:05.989 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:06.196 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:06.798 INFO:tasks.rados.rados.0.vpm137.stdout:3887: done (11 left) 2014-08-08T10:52:06.798 INFO:tasks.rados.rados.0.vpm137.stdout:3896: delete oid 461 current snap is 0 2014-08-08T10:52:06.799 INFO:tasks.rados.rados.0.vpm137.stdout:3886: finishing write tid 1 to vpm1375878-120 2014-08-08T10:52:06.799 INFO:tasks.rados.rados.0.vpm137.stdout:3886: finishing write tid 2 to vpm1375878-120 2014-08-08T10:52:06.799 INFO:tasks.rados.rados.0.vpm137.stdout:3886: finishing write tid 3 to vpm1375878-120 2014-08-08T10:52:06.799 INFO:tasks.rados.rados.0.vpm137.stdout:3883: expect (ObjNum 1498 snap 0 seq_num 1498) 2014-08-08T10:52:07.286 INFO:tasks.rados.rados.0.vpm137.stdout:3883: done (11 left) 2014-08-08T10:52:07.286 INFO:tasks.rados.rados.0.vpm137.stdout:3896: done (10 left) 2014-08-08T10:52:07.286 INFO:tasks.rados.rados.0.vpm137.stdout:3897: delete oid 139 current snap is 0 2014-08-08T10:52:07.287 INFO:tasks.rados.rados.0.vpm137.stdout:3886: finishing write tid 5 to vpm1375878-120 2014-08-08T10:52:07.287 INFO:tasks.rados.rados.0.vpm137.stdout:3886: finishing write tid 6 to vpm1375878-120 2014-08-08T10:52:07.287 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 120 v 432 (ObjNum 2378 snap 0 seq_num 2378) dirty exists 2014-08-08T10:52:07.288 INFO:tasks.rados.rados.0.vpm137.stdout:3882: expect (ObjNum 1919 snap 0 seq_num 1919) 2014-08-08T10:52:08.935 INFO:tasks.rados.rados.0.vpm137.stdout:3882: done (10 left) 2014-08-08T10:52:08.937 INFO:tasks.rados.rados.0.vpm137.stdout:3885: done (9 left) 2014-08-08T10:52:08.937 INFO:tasks.rados.rados.0.vpm137.stdout:3886: done (8 left) 2014-08-08T10:52:08.937 INFO:tasks.rados.rados.0.vpm137.stdout:3897: done (7 left) 2014-08-08T10:52:08.938 INFO:tasks.rados.rados.0.vpm137.stdout:3898: read oid 159 snap -1 2014-08-08T10:52:08.938 INFO:tasks.rados.rados.0.vpm137.stdout:3899: write oid 74 current snap is 0 2014-08-08T10:52:08.938 INFO:tasks.rados.rados.0.vpm137.stdout:3899: seq_num 2386 ranges {204267=1} 2014-08-08T10:52:08.941 INFO:tasks.rados.rados.0.vpm137.stdout:3899: writing vpm1375878-74 from 204267 to 204268 tid 1 2014-08-08T10:52:08.942 INFO:tasks.rados.rados.0.vpm137.stdout:3888: finishing write tid 1 to vpm1375878-396 2014-08-08T10:52:08.942 INFO:tasks.rados.rados.0.vpm137.stdout:3888: finishing write tid 2 to vpm1375878-396 2014-08-08T10:52:08.942 INFO:tasks.rados.rados.0.vpm137.stdout:3888: finishing write tid 3 to vpm1375878-396 2014-08-08T10:52:08.943 INFO:tasks.rados.rados.0.vpm137.stdout:3888: finishing write tid 5 to vpm1375878-396 2014-08-08T10:52:08.943 INFO:tasks.rados.rados.0.vpm137.stdout:3888: finishing write tid 6 to vpm1375878-396 2014-08-08T10:52:08.944 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 396 v 459 (ObjNum 2379 snap 0 seq_num 2379) dirty exists 2014-08-08T10:52:08.944 INFO:tasks.rados.rados.0.vpm137.stdout:3890: finishing write tid 1 to vpm1375878-408 2014-08-08T10:52:08.944 INFO:tasks.rados.rados.0.vpm137.stdout:3890: finishing write tid 3 to vpm1375878-408 2014-08-08T10:52:08.944 INFO:tasks.rados.rados.0.vpm137.stdout:3889: expect (ObjNum 2325 snap 0 seq_num 2325) 2014-08-08T10:52:09.220 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:09.935 INFO:tasks.rados.rados.0.vpm137.stdout:3888: done (8 left) 2014-08-08T10:52:09.935 INFO:tasks.rados.rados.0.vpm137.stdout:3889: done (7 left) 2014-08-08T10:52:09.935 INFO:tasks.rados.rados.0.vpm137.stdout:3900: read oid 205 snap -1 2014-08-08T10:52:09.936 INFO:tasks.rados.rados.0.vpm137.stdout:3901: read oid 109 snap -1 2014-08-08T10:52:09.936 INFO:tasks.rados.rados.0.vpm137.stdout:3902: read oid 488 snap -1 2014-08-08T10:52:09.936 INFO:tasks.rados.rados.0.vpm137.stdout:3903: delete oid 452 current snap is 0 2014-08-08T10:52:09.937 INFO:tasks.rados.rados.0.vpm137.stdout:3890: finishing write tid 4 to vpm1375878-408 2014-08-08T10:52:09.938 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 408 v 372 (ObjNum 2380 snap 0 seq_num 2380) dirty exists 2014-08-08T10:52:09.938 INFO:tasks.rados.rados.0.vpm137.stdout:3892: finishing write tid 1 to vpm1375878-387 2014-08-08T10:52:09.938 INFO:tasks.rados.rados.0.vpm137.stdout:3892: finishing write tid 3 to vpm1375878-387 2014-08-08T10:52:09.949 INFO:tasks.rados.rados.0.vpm137.stdout:3892: finishing write tid 4 to vpm1375878-387 2014-08-08T10:52:09.949 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 387 v 515 (ObjNum 2382 snap 0 seq_num 2382) dirty exists 2014-08-08T10:52:09.949 INFO:tasks.rados.rados.0.vpm137.stdout:3894: expect (ObjNum 2308 snap 0 seq_num 2308) 2014-08-08T10:52:10.179 INFO:tasks.rados.rados.0.vpm137.stdout:3890: done (10 left) 2014-08-08T10:52:10.179 INFO:tasks.rados.rados.0.vpm137.stdout:3892: done (9 left) 2014-08-08T10:52:10.179 INFO:tasks.rados.rados.0.vpm137.stdout:3894: done (8 left) 2014-08-08T10:52:10.180 INFO:tasks.rados.rados.0.vpm137.stdout:3903: done (7 left) 2014-08-08T10:52:10.180 INFO:tasks.rados.rados.0.vpm137.stdout:3904: read oid 37 snap -1 2014-08-08T10:52:10.180 INFO:tasks.rados.rados.0.vpm137.stdout:3905: delete oid 92 current snap is 0 2014-08-08T10:52:10.181 INFO:tasks.rados.rados.0.vpm137.stdout:3893: expect (ObjNum 2358 snap 0 seq_num 2358) 2014-08-08T10:52:10.841 INFO:tasks.rados.rados.0.vpm137.stdout:3893: done (8 left) 2014-08-08T10:52:10.841 INFO:tasks.rados.rados.0.vpm137.stdout:3905: done (7 left) 2014-08-08T10:52:10.842 INFO:tasks.rados.rados.0.vpm137.stdout:3906: write oid 250 current snap is 0 2014-08-08T10:52:10.842 INFO:tasks.rados.rados.0.vpm137.stdout:3906: seq_num 2389 ranges {588134=689888,1388133=1} 2014-08-08T10:52:10.872 INFO:tasks.rados.rados.0.vpm137.stdout:3906: writing vpm1375878-250 from 588134 to 1278022 tid 1 2014-08-08T10:52:10.892 INFO:tasks.rados.rados.0.vpm137.stdout:3906: writing vpm1375878-250 from 1388133 to 1388134 tid 2 2014-08-08T10:52:10.892 INFO:tasks.rados.rados.0.vpm137.stdout:3895: finishing write tid 1 to vpm1375878-232 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:3895: finishing write tid 2 to vpm1375878-232 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:3895: finishing write tid 3 to vpm1375878-232 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:3895: finishing write tid 5 to vpm1375878-232 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:3895: finishing write tid 6 to vpm1375878-232 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 232 v 641 (ObjNum 2383 snap 0 seq_num 2383) dirty exists 2014-08-08T10:52:10.893 INFO:tasks.rados.rados.0.vpm137.stdout:3898: expect (ObjNum 2274 snap 0 seq_num 2274) 2014-08-08T10:52:12.795 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:12.836 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:12.946 INFO:tasks.rados.rados.0.vpm137.stdout:3895: done (7 left) 2014-08-08T10:52:12.946 INFO:tasks.rados.rados.0.vpm137.stdout:3898: done (6 left) 2014-08-08T10:52:12.947 INFO:tasks.rados.rados.0.vpm137.stdout:3907: write oid 434 current snap is 0 2014-08-08T10:52:12.947 INFO:tasks.rados.rados.0.vpm137.stdout:3907: seq_num 2390 ranges {799715=546364,1599714=1} 2014-08-08T10:52:12.980 INFO:tasks.rados.rados.0.vpm137.stdout:3907: writing vpm1375878-434 from 799715 to 1346079 tid 1 2014-08-08T10:52:12.985 INFO:tasks.rados.rados.0.vpm137.stdout:3907: writing vpm1375878-434 from 1599714 to 1599715 tid 2 2014-08-08T10:52:12.986 INFO:tasks.rados.rados.0.vpm137.stdout:3899: finishing write tid 1 to vpm1375878-74 2014-08-08T10:52:12.986 INFO:tasks.rados.rados.0.vpm137.stdout:3899: finishing write tid 3 to vpm1375878-74 2014-08-08T10:52:12.987 INFO:tasks.rados.rados.0.vpm137.stdout:3899: finishing write tid 4 to vpm1375878-74 2014-08-08T10:52:12.987 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 74 v 461 (ObjNum 2386 snap 0 seq_num 2386) dirty exists 2014-08-08T10:52:12.987 INFO:tasks.rados.rados.0.vpm137.stdout:3901: expect (ObjNum 1424 snap 0 seq_num 1424) 2014-08-08T10:52:13.002 INFO:tasks.rados.rados.0.vpm137.stdout:3899: done (6 left) 2014-08-08T10:52:13.002 INFO:tasks.rados.rados.0.vpm137.stdout:3901: done (5 left) 2014-08-08T10:52:13.002 INFO:tasks.rados.rados.0.vpm137.stdout:3908: read oid 13 snap -1 2014-08-08T10:52:13.003 INFO:tasks.rados.rados.0.vpm137.stdout:3909: read oid 275 snap -1 2014-08-08T10:52:13.003 INFO:tasks.rados.rados.0.vpm137.stdout:3910: write oid 53 current snap is 0 2014-08-08T10:52:13.004 INFO:tasks.rados.rados.0.vpm137.stdout:3910: seq_num 2391 ranges {495969=669592,1295968=1} 2014-08-08T10:52:13.031 INFO:tasks.rados.rados.0.vpm137.stdout:3910: writing vpm1375878-53 from 495969 to 1165561 tid 1 2014-08-08T10:52:13.033 INFO:tasks.rados.rados.0.vpm137.stdout:3910: writing vpm1375878-53 from 1295968 to 1295969 tid 2 2014-08-08T10:52:13.034 INFO:tasks.rados.rados.0.vpm137.stdout:3900: expect (ObjNum 2098 snap 0 seq_num 2098) 2014-08-08T10:52:13.283 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:51:13.454007', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89725824, u'health': u'HEALTH_OK', u'kb_used': 9227744, u'store_stats': {u'bytes_total': 25719043, u'bytes_log': 6297183, u'last_updated': u'0.000000', u'bytes_misc': 19421860, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:51:17.562465', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89724836, u'health': u'HEALTH_OK', u'kb_used': 9228732, u'store_stats': {u'bytes_total': 28906859, u'bytes_log': 9485055, u'last_updated': u'0.000000', u'bytes_misc': 19421804, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 849582080, u'bytes_used': 1700040704, u'op_per_sec': 9, u'version': 1698, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1882622, u'write_bytes_sec': 418667, u'bytes_avail': 427586973696}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:13.285 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:13.402 INFO:tasks.rados.rados.0.vpm137.stdout:3900: done (7 left) 2014-08-08T10:52:13.402 INFO:tasks.rados.rados.0.vpm137.stdout:3911: write oid 245 current snap is 0 2014-08-08T10:52:13.403 INFO:tasks.rados.rados.0.vpm137.stdout:3911: seq_num 2392 ranges {439775=641397,1778347=261428} 2014-08-08T10:52:13.427 INFO:tasks.rados.rados.0.vpm137.stdout:3911: writing vpm1375878-245 from 439775 to 1081172 tid 1 2014-08-08T10:52:13.444 INFO:tasks.rados.rados.0.vpm137.stdout:3911: writing vpm1375878-245 from 1778347 to 2039775 tid 2 2014-08-08T10:52:13.445 INFO:tasks.rados.rados.0.vpm137.stdout:3902: expect (ObjNum 1796 snap 0 seq_num 1796) 2014-08-08T10:52:13.723 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:13.724 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:14.048 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:14.464 INFO:tasks.rados.rados.0.vpm137.stdout:3902: done (7 left) 2014-08-08T10:52:14.465 INFO:tasks.rados.rados.0.vpm137.stdout:3912: write oid 31 current snap is 0 2014-08-08T10:52:14.465 INFO:tasks.rados.rados.0.vpm137.stdout:3912: seq_num 2393 ranges {784605=764625,2292479=492126} 2014-08-08T10:52:14.494 INFO:tasks.rados.rados.0.vpm137.stdout:3912: writing vpm1375878-31 from 784605 to 1549230 tid 1 2014-08-08T10:52:14.520 INFO:tasks.rados.rados.0.vpm137.stdout:3912: writing vpm1375878-31 from 2292479 to 2784605 tid 2 2014-08-08T10:52:14.521 INFO:tasks.rados.rados.0.vpm137.stdout:3904: expect (ObjNum 2243 snap 0 seq_num 2243) 2014-08-08T10:52:15.102 INFO:tasks.rados.rados.0.vpm137.stdout:3904: done (7 left) 2014-08-08T10:52:15.103 INFO:tasks.rados.rados.0.vpm137.stdout:3913: write oid 148 current snap is 0 2014-08-08T10:52:15.103 INFO:tasks.rados.rados.0.vpm137.stdout:3913: seq_num 2394 ranges {597210=469677,1856210=434141,2700363=296847} 2014-08-08T10:52:15.122 INFO:tasks.rados.rados.0.vpm137.stdout:3913: writing vpm1375878-148 from 597210 to 1066887 tid 1 2014-08-08T10:52:15.144 INFO:tasks.rados.rados.0.vpm137.stdout:3913: writing vpm1375878-148 from 1856210 to 2290351 tid 2 2014-08-08T10:52:15.159 INFO:tasks.rados.rados.0.vpm137.stdout:3913: writing vpm1375878-148 from 2700363 to 2997210 tid 3 2014-08-08T10:52:15.161 INFO:tasks.rados.rados.0.vpm137.stdout:3906: finishing write tid 1 to vpm1375878-250 2014-08-08T10:52:15.161 INFO:tasks.rados.rados.0.vpm137.stdout:3906: finishing write tid 2 to vpm1375878-250 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:3906: finishing write tid 4 to vpm1375878-250 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:3906: finishing write tid 5 to vpm1375878-250 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 250 v 435 (ObjNum 2389 snap 0 seq_num 2389) dirty exists 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:3907: finishing write tid 1 to vpm1375878-434 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:3907: finishing write tid 2 to vpm1375878-434 2014-08-08T10:52:15.162 INFO:tasks.rados.rados.0.vpm137.stdout:3907: finishing write tid 4 to vpm1375878-434 2014-08-08T10:52:15.163 INFO:tasks.rados.rados.0.vpm137.stdout:3907: finishing write tid 5 to vpm1375878-434 2014-08-08T10:52:15.169 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 434 v 412 (ObjNum 2390 snap 0 seq_num 2390) dirty exists 2014-08-08T10:52:15.169 INFO:tasks.rados.rados.0.vpm137.stdout:3908: expect (ObjNum 1896 snap 0 seq_num 1896) 2014-08-08T10:52:16.855 INFO:tasks.rados.rados.0.vpm137.stdout:3906: done (7 left) 2014-08-08T10:52:16.855 INFO:tasks.rados.rados.0.vpm137.stdout:3907: done (6 left) 2014-08-08T10:52:16.855 INFO:tasks.rados.rados.0.vpm137.stdout:3908: done (5 left) 2014-08-08T10:52:16.856 INFO:tasks.rados.rados.0.vpm137.stdout:3914: read oid 124 snap -1 2014-08-08T10:52:16.856 INFO:tasks.rados.rados.0.vpm137.stdout:3915: read oid 268 snap -1 2014-08-08T10:52:16.857 INFO:tasks.rados.rados.0.vpm137.stdout:3916: write oid 450 current snap is 0 2014-08-08T10:52:16.857 INFO:tasks.rados.rados.0.vpm137.stdout:3916: seq_num 2395 ranges {541016=592906,1534247=539408,2500565=611270,3554055=186961} 2014-08-08T10:52:16.891 INFO:tasks.rados.rados.0.vpm137.stdout:3916: writing vpm1375878-450 from 541016 to 1133922 tid 1 2014-08-08T10:52:16.930 INFO:tasks.rados.rados.0.vpm137.stdout:3916: writing vpm1375878-450 from 1534247 to 2073655 tid 2 2014-08-08T10:52:16.970 INFO:tasks.rados.rados.0.vpm137.stdout:3916: writing vpm1375878-450 from 2500565 to 3111835 tid 3 2014-08-08T10:52:16.980 INFO:tasks.rados.rados.0.vpm137.stdout:3916: writing vpm1375878-450 from 3554055 to 3741016 tid 4 2014-08-08T10:52:16.981 INFO:tasks.rados.rados.0.vpm137.stdout:3910: finishing write tid 1 to vpm1375878-53 2014-08-08T10:52:16.981 INFO:tasks.rados.rados.0.vpm137.stdout:3910: finishing write tid 2 to vpm1375878-53 2014-08-08T10:52:16.981 INFO:tasks.rados.rados.0.vpm137.stdout:3910: finishing write tid 4 to vpm1375878-53 2014-08-08T10:52:16.982 INFO:tasks.rados.rados.0.vpm137.stdout:3910: finishing write tid 5 to vpm1375878-53 2014-08-08T10:52:16.982 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 53 v 322 (ObjNum 2391 snap 0 seq_num 2391) dirty exists 2014-08-08T10:52:16.982 INFO:tasks.rados.rados.0.vpm137.stdout:3911: finishing write tid 1 to vpm1375878-245 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:3911: finishing write tid 2 to vpm1375878-245 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:3911: finishing write tid 4 to vpm1375878-245 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:3911: finishing write tid 5 to vpm1375878-245 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 245 v 375 (ObjNum 2392 snap 0 seq_num 2392) dirty exists 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:3912: finishing write tid 1 to vpm1375878-31 2014-08-08T10:52:16.983 INFO:tasks.rados.rados.0.vpm137.stdout:3912: finishing write tid 2 to vpm1375878-31 2014-08-08T10:52:16.984 INFO:tasks.rados.rados.0.vpm137.stdout:3912: finishing write tid 4 to vpm1375878-31 2014-08-08T10:52:16.984 INFO:tasks.rados.rados.0.vpm137.stdout:3912: finishing write tid 5 to vpm1375878-31 2014-08-08T10:52:16.984 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 31 v 446 (ObjNum 2393 snap 0 seq_num 2393) dirty exists 2014-08-08T10:52:16.984 INFO:tasks.rados.rados.0.vpm137.stdout:3913: finishing write tid 1 to vpm1375878-148 2014-08-08T10:52:16.985 INFO:tasks.rados.rados.0.vpm137.stdout:3913: finishing write tid 2 to vpm1375878-148 2014-08-08T10:52:16.985 INFO:tasks.rados.rados.0.vpm137.stdout:3913: finishing write tid 3 to vpm1375878-148 2014-08-08T10:52:16.985 INFO:tasks.rados.rados.0.vpm137.stdout:3913: finishing write tid 5 to vpm1375878-148 2014-08-08T10:52:16.985 INFO:tasks.rados.rados.0.vpm137.stdout:3913: finishing write tid 6 to vpm1375878-148 2014-08-08T10:52:16.985 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 148 v 462 (ObjNum 2394 snap 0 seq_num 2394) dirty exists 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 1 to vpm1375878-450 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3909: done (7 left) 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3910: done (6 left) 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3911: done (5 left) 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3912: done (4 left) 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3913: done (3 left) 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3917: read oid 201 snap -1 2014-08-08T10:52:16.986 INFO:tasks.rados.rados.0.vpm137.stdout:3918: read oid 353 snap -1 2014-08-08T10:52:16.987 INFO:tasks.rados.rados.0.vpm137.stdout:3919: read oid 106 snap -1 2014-08-08T10:52:16.987 INFO:tasks.rados.rados.0.vpm137.stdout:3920: read oid 85 snap -1 2014-08-08T10:52:16.987 INFO:tasks.rados.rados.0.vpm137.stdout:3921: write oid 492 current snap is 0 2014-08-08T10:52:16.987 INFO:tasks.rados.rados.0.vpm137.stdout:3921: seq_num 2396 ranges {485845=1} 2014-08-08T10:52:16.990 INFO:tasks.rados.rados.0.vpm137.stdout:3921: writing vpm1375878-492 from 485845 to 485846 tid 1 2014-08-08T10:52:16.990 INFO:tasks.rados.rados.0.vpm137.stdout:3922: read oid 340 snap -1 2014-08-08T10:52:16.990 INFO:tasks.rados.rados.0.vpm137.stdout:3923: read oid 65 snap -1 2014-08-08T10:52:16.991 INFO:tasks.rados.rados.0.vpm137.stdout:3924: read oid 253 snap -1 2014-08-08T10:52:16.991 INFO:tasks.rados.rados.0.vpm137.stdout:3925: read oid 107 snap -1 2014-08-08T10:52:16.991 INFO:tasks.rados.rados.0.vpm137.stdout:3926: write oid 385 current snap is 0 2014-08-08T10:52:16.992 INFO:tasks.rados.rados.0.vpm137.stdout:3926: seq_num 2397 ranges {180051=1} 2014-08-08T10:52:16.993 INFO:tasks.rados.rados.0.vpm137.stdout:3926: writing vpm1375878-385 from 180051 to 180052 tid 1 2014-08-08T10:52:16.994 INFO:tasks.rados.rados.0.vpm137.stdout:3927: read oid 40 snap -1 2014-08-08T10:52:16.994 INFO:tasks.rados.rados.0.vpm137.stdout:3928: write oid 442 current snap is 0 2014-08-08T10:52:16.994 INFO:tasks.rados.rados.0.vpm137.stdout:3928: seq_num 2398 ranges {524881=400000} 2014-08-08T10:52:17.013 INFO:tasks.rados.rados.0.vpm137.stdout:3928: writing vpm1375878-442 from 524881 to 924881 tid 1 2014-08-08T10:52:17.014 INFO:tasks.rados.rados.0.vpm137.stdout:3929: read oid 135 snap -1 2014-08-08T10:52:17.014 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:17.120 INFO:tasks.rados.rados.0.vpm137.stdout:3914: expect (ObjNum 2262 snap 0 seq_num 2262) 2014-08-08T10:52:17.326 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:19.151 INFO:tasks.rados.rados.0.vpm137.stdout:3914: done (15 left) 2014-08-08T10:52:19.152 INFO:tasks.rados.rados.0.vpm137.stdout:3930: write oid 432 current snap is 0 2014-08-08T10:52:19.152 INFO:tasks.rados.rados.0.vpm137.stdout:3930: seq_num 2399 ranges {586063=668667,1803255=382808} 2014-08-08T10:52:19.192 INFO:tasks.rados.rados.0.vpm137.stdout:3930: writing vpm1375878-432 from 586063 to 1254730 tid 1 2014-08-08T10:52:19.226 INFO:tasks.rados.rados.0.vpm137.stdout:3930: writing vpm1375878-432 from 1803255 to 2186063 tid 2 2014-08-08T10:52:19.228 INFO:tasks.rados.rados.0.vpm137.stdout:3915: expect (ObjNum 1630 snap 0 seq_num 1630) 2014-08-08T10:52:19.930 INFO:tasks.rados.rados.0.vpm137.stdout:3915: done (15 left) 2014-08-08T10:52:19.930 INFO:tasks.rados.rados.0.vpm137.stdout:3931: read oid 80 snap -1 2014-08-08T10:52:19.931 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:19.931 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 2 to vpm1375878-450 2014-08-08T10:52:19.932 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 3 to vpm1375878-450 2014-08-08T10:52:19.932 INFO:tasks.rados.rados.0.vpm137.stdout:3917: expect (ObjNum 2351 snap 0 seq_num 2351) 2014-08-08T10:52:20.600 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:20.653 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:20.931 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 849855848, u'bytes_used': 1696538624, u'op_per_sec': 13, u'version': 1700, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2902670, u'write_bytes_sec': 864777, u'bytes_avail': 427590475776}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:20.932 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:21.178 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:21.179 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:21.433 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:21.928 INFO:tasks.rados.rados.0.vpm137.stdout:3917: done (15 left) 2014-08-08T10:52:21.928 INFO:tasks.rados.rados.0.vpm137.stdout:3932: write oid 209 current snap is 0 2014-08-08T10:52:21.929 INFO:tasks.rados.rados.0.vpm137.stdout:3932: seq_num 2400 ranges {681293=640471,1881292=1} 2014-08-08T10:52:21.967 INFO:tasks.rados.rados.0.vpm137.stdout:3932: writing vpm1375878-209 from 681293 to 1321764 tid 1 2014-08-08T10:52:21.982 INFO:tasks.rados.rados.0.vpm137.stdout:3932: writing vpm1375878-209 from 1881292 to 1881293 tid 2 2014-08-08T10:52:21.983 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 4 to vpm1375878-450 2014-08-08T10:52:21.983 INFO:tasks.rados.rados.0.vpm137.stdout:3919: expect (ObjNum 2233 snap 0 seq_num 2233) 2014-08-08T10:52:22.018 INFO:tasks.rados.rados.0.vpm137.stdout:3918: done (15 left) 2014-08-08T10:52:22.018 INFO:tasks.rados.rados.0.vpm137.stdout:3919: done (14 left) 2014-08-08T10:52:22.019 INFO:tasks.rados.rados.0.vpm137.stdout:3933: read oid 429 snap -1 2014-08-08T10:52:22.019 INFO:tasks.rados.rados.0.vpm137.stdout:3934: write oid 273 current snap is 0 2014-08-08T10:52:22.019 INFO:tasks.rados.rados.0.vpm137.stdout:3934: seq_num 2401 ranges {626123=764724,1996448=629675} 2014-08-08T10:52:22.060 INFO:tasks.rados.rados.0.vpm137.stdout:3934: writing vpm1375878-273 from 626123 to 1390847 tid 1 2014-08-08T10:52:22.097 INFO:tasks.rados.rados.0.vpm137.stdout:3934: writing vpm1375878-273 from 1996448 to 2626123 tid 2 2014-08-08T10:52:22.099 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 6 to vpm1375878-450 2014-08-08T10:52:22.099 INFO:tasks.rados.rados.0.vpm137.stdout:3916: finishing write tid 7 to vpm1375878-450 2014-08-08T10:52:22.100 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 450 v 341 (ObjNum 2395 snap 0 seq_num 2395) dirty exists 2014-08-08T10:52:22.100 INFO:tasks.rados.rados.0.vpm137.stdout:3920: expect (ObjNum 2090 snap 0 seq_num 2090) 2014-08-08T10:52:23.645 INFO:tasks.rados.rados.0.vpm137.stdout:3916: done (15 left) 2014-08-08T10:52:23.645 INFO:tasks.rados.rados.0.vpm137.stdout:3920: done (14 left) 2014-08-08T10:52:23.645 INFO:tasks.rados.rados.0.vpm137.stdout:3935: write oid 302 current snap is 0 2014-08-08T10:52:23.645 INFO:tasks.rados.rados.0.vpm137.stdout:3935: seq_num 2402 ranges {569928=620176,1958131=533622,2904469=465459} 2014-08-08T10:52:23.669 INFO:tasks.rados.rados.0.vpm137.stdout:3935: writing vpm1375878-302 from 569928 to 1190104 tid 1 2014-08-08T10:52:23.698 INFO:tasks.rados.rados.0.vpm137.stdout:3935: writing vpm1375878-302 from 1958131 to 2491753 tid 2 2014-08-08T10:52:23.719 INFO:tasks.rados.rados.0.vpm137.stdout:3935: writing vpm1375878-302 from 2904469 to 3369928 tid 3 2014-08-08T10:52:23.721 INFO:tasks.rados.rados.0.vpm137.stdout:3921: finishing write tid 1 to vpm1375878-492 2014-08-08T10:52:23.722 INFO:tasks.rados.rados.0.vpm137.stdout:3921: finishing write tid 3 to vpm1375878-492 2014-08-08T10:52:23.722 INFO:tasks.rados.rados.0.vpm137.stdout:3924: expect (ObjNum 2260 snap 0 seq_num 2260) 2014-08-08T10:52:24.411 INFO:tasks.rados.rados.0.vpm137.stdout:3924: done (14 left) 2014-08-08T10:52:24.412 INFO:tasks.rados.rados.0.vpm137.stdout:3936: read oid 400 snap -1 2014-08-08T10:52:24.412 INFO:tasks.rados.rados.0.vpm137.stdout:3937: write oid 132 current snap is 0 2014-08-08T10:52:24.412 INFO:tasks.rados.rados.0.vpm137.stdout:3937: seq_num 2403 ranges {782534=743404,1940039=755242,3159967=422567} 2014-08-08T10:52:24.458 INFO:tasks.rados.rados.0.vpm137.stdout:3937: writing vpm1375878-132 from 782534 to 1525938 tid 1 2014-08-08T10:52:24.464 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:24.494 INFO:tasks.rados.rados.0.vpm137.stdout:3937: writing vpm1375878-132 from 1940039 to 2695281 tid 2 2014-08-08T10:52:24.513 INFO:tasks.rados.rados.0.vpm137.stdout:3937: writing vpm1375878-132 from 3159967 to 3582534 tid 3 2014-08-08T10:52:24.515 INFO:tasks.rados.rados.0.vpm137.stdout:3923: expect (ObjNum 2155 snap 0 seq_num 2155) 2014-08-08T10:52:24.756 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:24.800 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:25.799 INFO:tasks.rados.rados.0.vpm137.stdout:3923: done (15 left) 2014-08-08T10:52:25.800 INFO:tasks.rados.rados.0.vpm137.stdout:3938: write oid 129 current snap is 0 2014-08-08T10:52:25.800 INFO:tasks.rados.rados.0.vpm137.stdout:3938: seq_num 2404 ranges {327363=1} 2014-08-08T10:52:25.803 INFO:tasks.rados.rados.0.vpm137.stdout:3938: writing vpm1375878-129 from 327363 to 327364 tid 1 2014-08-08T10:52:25.805 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:25.806 INFO:tasks.rados.rados.0.vpm137.stdout:3922: expect (ObjNum 2095 snap 0 seq_num 2095) 2014-08-08T10:52:28.062 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 853403845, u'bytes_used': 1696538624, u'op_per_sec': 12, u'version': 1701, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2702625, u'write_bytes_sec': 895353, u'bytes_avail': 427590475776}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:28.062 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:28.398 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:28.398 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:28.665 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:30.621 INFO:tasks.rados.rados.0.vpm137.stdout:3922: done (15 left) 2014-08-08T10:52:30.628 INFO:tasks.rados.rados.0.vpm137.stdout:3939: read oid 284 snap -1 2014-08-08T10:52:30.629 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 16 2014-08-08T10:52:30.629 INFO:tasks.rados.rados.0.vpm137.stdout:3929: expect (ObjNum 2186 snap 0 seq_num 2186) 2014-08-08T10:52:30.704 INFO:tasks.rados.rados.0.vpm137.stdout:3929: done (15 left) 2014-08-08T10:52:30.704 INFO:tasks.rados.rados.0.vpm137.stdout:3940: delete oid 119 current snap is 0 2014-08-08T10:52:30.705 INFO:tasks.rados.rados.0.vpm137.stdout:3921: finishing write tid 4 to vpm1375878-492 2014-08-08T10:52:30.705 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 492 v 377 (ObjNum 2396 snap 0 seq_num 2396) dirty exists 2014-08-08T10:52:30.705 INFO:tasks.rados.rados.0.vpm137.stdout:3926: finishing write tid 1 to vpm1375878-385 2014-08-08T10:52:30.706 INFO:tasks.rados.rados.0.vpm137.stdout:3926: finishing write tid 3 to vpm1375878-385 2014-08-08T10:52:30.706 INFO:tasks.rados.rados.0.vpm137.stdout:3926: finishing write tid 4 to vpm1375878-385 2014-08-08T10:52:30.706 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 385 v 448 (ObjNum 2397 snap 0 seq_num 2397) dirty exists 2014-08-08T10:52:30.706 INFO:tasks.rados.rados.0.vpm137.stdout:3925: expect (ObjNum 2264 snap 0 seq_num 2264) 2014-08-08T10:52:31.701 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:31.932 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:31.978 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:33.505 INFO:tasks.rados.rados.0.vpm137.stdout:3921: done (15 left) 2014-08-08T10:52:33.505 INFO:tasks.rados.rados.0.vpm137.stdout:3925: done (14 left) 2014-08-08T10:52:33.505 INFO:tasks.rados.rados.0.vpm137.stdout:3926: done (13 left) 2014-08-08T10:52:33.505 INFO:tasks.rados.rados.0.vpm137.stdout:3940: done (12 left) 2014-08-08T10:52:33.506 INFO:tasks.rados.rados.0.vpm137.stdout:3941: read oid 73 snap -1 2014-08-08T10:52:33.506 INFO:tasks.rados.rados.0.vpm137.stdout:3942: delete oid 67 current snap is 0 2014-08-08T10:52:33.506 INFO:tasks.rados.rados.0.vpm137.stdout:3927: expect (ObjNum 1893 snap 0 seq_num 1893) 2014-08-08T10:52:34.389 INFO:tasks.rados.rados.0.vpm137.stdout:3927: done (13 left) 2014-08-08T10:52:34.390 INFO:tasks.rados.rados.0.vpm137.stdout:3942: done (12 left) 2014-08-08T10:52:34.391 INFO:tasks.rados.rados.0.vpm137.stdout:3943: read oid 247 snap -1 2014-08-08T10:52:34.391 INFO:tasks.rados.rados.0.vpm137.stdout:3944: write oid 74 current snap is 0 2014-08-08T10:52:34.391 INFO:tasks.rados.rados.0.vpm137.stdout:3944: seq_num 2407 ranges {710205=666718,1510204=1} 2014-08-08T10:52:34.431 INFO:tasks.rados.rados.0.vpm137.stdout:3944: writing vpm1375878-74 from 710205 to 1376923 tid 1 2014-08-08T10:52:34.443 INFO:tasks.rados.rados.0.vpm137.stdout:3944: writing vpm1375878-74 from 1510204 to 1510205 tid 2 2014-08-08T10:52:34.443 INFO:tasks.rados.rados.0.vpm137.stdout:3928: finishing write tid 1 to vpm1375878-442 2014-08-08T10:52:34.444 INFO:tasks.rados.rados.0.vpm137.stdout:3928: finishing write tid 3 to vpm1375878-442 2014-08-08T10:52:34.444 INFO:tasks.rados.rados.0.vpm137.stdout:3928: finishing write tid 4 to vpm1375878-442 2014-08-08T10:52:34.445 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 442 v 324 (ObjNum 2398 snap 0 seq_num 2398) dirty exists 2014-08-08T10:52:34.446 INFO:tasks.rados.rados.0.vpm137.stdout:3930: finishing write tid 1 to vpm1375878-432 2014-08-08T10:52:34.446 INFO:tasks.rados.rados.0.vpm137.stdout:3930: finishing write tid 2 to vpm1375878-432 2014-08-08T10:52:34.447 INFO:tasks.rados.rados.0.vpm137.stdout:3930: finishing write tid 4 to vpm1375878-432 2014-08-08T10:52:34.447 INFO:tasks.rados.rados.0.vpm137.stdout:3930: finishing write tid 5 to vpm1375878-432 2014-08-08T10:52:34.447 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 432 v 450 (ObjNum 2399 snap 0 seq_num 2399) dirty exists 2014-08-08T10:52:34.447 INFO:tasks.rados.rados.0.vpm137.stdout:3931: expect (ObjNum 1801 snap 0 seq_num 1801) 2014-08-08T10:52:34.536 INFO:tasks.rados.rados.0.vpm137.stdout:3928: done (13 left) 2014-08-08T10:52:34.536 INFO:tasks.rados.rados.0.vpm137.stdout:3930: done (12 left) 2014-08-08T10:52:34.537 INFO:tasks.rados.rados.0.vpm137.stdout:3931: done (11 left) 2014-08-08T10:52:34.537 INFO:tasks.rados.rados.0.vpm137.stdout:3945: read oid 489 snap -1 2014-08-08T10:52:34.537 INFO:tasks.rados.rados.0.vpm137.stdout:3946: read oid 161 snap -1 2014-08-08T10:52:34.538 INFO:tasks.rados.rados.0.vpm137.stdout:3947: write oid 433 current snap is 0 2014-08-08T10:52:34.538 INFO:tasks.rados.rados.0.vpm137.stdout:3947: seq_num 2408 ranges {522811=639546,1722810=1} 2014-08-08T10:52:34.599 INFO:tasks.rados.rados.0.vpm137.stdout:3947: writing vpm1375878-433 from 522811 to 1162357 tid 1 2014-08-08T10:52:34.608 INFO:tasks.rados.rados.0.vpm137.stdout:3947: writing vpm1375878-433 from 1722810 to 1722811 tid 2 2014-08-08T10:52:34.610 INFO:tasks.rados.rados.0.vpm137.stdout:3932: finishing write tid 1 to vpm1375878-209 2014-08-08T10:52:34.610 INFO:tasks.rados.rados.0.vpm137.stdout:3932: finishing write tid 2 to vpm1375878-209 2014-08-08T10:52:34.610 INFO:tasks.rados.rados.0.vpm137.stdout:3932: finishing write tid 4 to vpm1375878-209 2014-08-08T10:52:34.611 INFO:tasks.rados.rados.0.vpm137.stdout:3932: finishing write tid 5 to vpm1375878-209 2014-08-08T10:52:34.613 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 209 v 518 (ObjNum 2400 snap 0 seq_num 2400) dirty exists 2014-08-08T10:52:34.613 INFO:tasks.rados.rados.0.vpm137.stdout:3934: finishing write tid 1 to vpm1375878-273 2014-08-08T10:52:34.613 INFO:tasks.rados.rados.0.vpm137.stdout:3934: finishing write tid 2 to vpm1375878-273 2014-08-08T10:52:34.613 INFO:tasks.rados.rados.0.vpm137.stdout:3934: finishing write tid 4 to vpm1375878-273 2014-08-08T10:52:34.614 INFO:tasks.rados.rados.0.vpm137.stdout:3934: finishing write tid 5 to vpm1375878-273 2014-08-08T10:52:34.614 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 273 v 432 (ObjNum 2401 snap 0 seq_num 2401) dirty exists 2014-08-08T10:52:34.614 INFO:tasks.rados.rados.0.vpm137.stdout:3935: finishing write tid 1 to vpm1375878-302 2014-08-08T10:52:34.615 INFO:tasks.rados.rados.0.vpm137.stdout:3935: finishing write tid 2 to vpm1375878-302 2014-08-08T10:52:34.615 INFO:tasks.rados.rados.0.vpm137.stdout:3935: finishing write tid 3 to vpm1375878-302 2014-08-08T10:52:34.615 INFO:tasks.rados.rados.0.vpm137.stdout:3935: finishing write tid 5 to vpm1375878-302 2014-08-08T10:52:34.616 INFO:tasks.rados.rados.0.vpm137.stdout:3935: finishing write tid 6 to vpm1375878-302 2014-08-08T10:52:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 302 v 645 (ObjNum 2402 snap 0 seq_num 2402) dirty exists 2014-08-08T10:52:34.617 INFO:tasks.rados.rados.0.vpm137.stdout:3936: expect (ObjNum 2275 snap 0 seq_num 2275) 2014-08-08T10:52:35.111 INFO:tasks.rados.rados.0.vpm137.stdout:3932: done (13 left) 2014-08-08T10:52:35.111 INFO:tasks.rados.rados.0.vpm137.stdout:3933: done (12 left) 2014-08-08T10:52:35.111 INFO:tasks.rados.rados.0.vpm137.stdout:3934: done (11 left) 2014-08-08T10:52:35.112 INFO:tasks.rados.rados.0.vpm137.stdout:3935: done (10 left) 2014-08-08T10:52:35.113 INFO:tasks.rados.rados.0.vpm137.stdout:3936: done (9 left) 2014-08-08T10:52:35.113 INFO:tasks.rados.rados.0.vpm137.stdout:3948: read oid 399 snap -1 2014-08-08T10:52:35.113 INFO:tasks.rados.rados.0.vpm137.stdout:3949: write oid 160 current snap is 0 2014-08-08T10:52:35.114 INFO:tasks.rados.rados.0.vpm137.stdout:3949: seq_num 2409 ranges {716216=762774,1946944=411485,2792024=724192} 2014-08-08T10:52:35.157 INFO:tasks.rados.rados.0.vpm137.stdout:3949: writing vpm1375878-160 from 716216 to 1478990 tid 1 2014-08-08T10:52:35.207 INFO:tasks.rados.rados.0.vpm137.stdout:3949: writing vpm1375878-160 from 1946944 to 2358429 tid 2 2014-08-08T10:52:35.257 INFO:tasks.rados.rados.0.vpm137.stdout:3949: writing vpm1375878-160 from 2792024 to 3516216 tid 3 2014-08-08T10:52:35.260 INFO:tasks.rados.rados.0.vpm137.stdout:3937: finishing write tid 1 to vpm1375878-132 2014-08-08T10:52:35.261 INFO:tasks.rados.rados.0.vpm137.stdout:3937: finishing write tid 2 to vpm1375878-132 2014-08-08T10:52:35.262 INFO:tasks.rados.rados.0.vpm137.stdout:3937: finishing write tid 3 to vpm1375878-132 2014-08-08T10:52:35.262 INFO:tasks.rados.rados.0.vpm137.stdout:3937: finishing write tid 5 to vpm1375878-132 2014-08-08T10:52:35.262 INFO:tasks.rados.rados.0.vpm137.stdout:3937: finishing write tid 6 to vpm1375878-132 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 132 v 466 (ObjNum 2403 snap 0 seq_num 2403) dirty exists 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:3938: finishing write tid 1 to vpm1375878-129 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:3938: finishing write tid 3 to vpm1375878-129 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:3938: finishing write tid 4 to vpm1375878-129 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 129 v 475 (ObjNum 2404 snap 0 seq_num 2404) dirty exists 2014-08-08T10:52:35.263 INFO:tasks.rados.rados.0.vpm137.stdout:3939: expect (ObjNum 2306 snap 0 seq_num 2306) 2014-08-08T10:52:35.291 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 853552065, u'bytes_used': 1699934208, u'op_per_sec': 3, u'version': 1702, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 158154, u'write_bytes_sec': 557890, u'bytes_avail': 427587080192}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:35.292 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:37.840 INFO:tasks.rados.rados.0.vpm137.stdout:3937: done (10 left) 2014-08-08T10:52:37.840 INFO:tasks.rados.rados.0.vpm137.stdout:3938: done (9 left) 2014-08-08T10:52:37.840 INFO:tasks.rados.rados.0.vpm137.stdout:3939: done (8 left) 2014-08-08T10:52:37.841 INFO:tasks.rados.rados.0.vpm137.stdout:3950: read oid 306 snap -1 2014-08-08T10:52:37.841 INFO:tasks.rados.rados.0.vpm137.stdout:3951: write oid 36 current snap is 0 2014-08-08T10:52:37.841 INFO:tasks.rados.rados.0.vpm137.stdout:3951: seq_num 2410 ranges {411446=618227,1660053=633104,2992226=219220} 2014-08-08T10:52:37.884 INFO:tasks.rados.rados.0.vpm137.stdout:3951: writing vpm1375878-36 from 411446 to 1029673 tid 1 2014-08-08T10:52:37.931 INFO:tasks.rados.rados.0.vpm137.stdout:3951: writing vpm1375878-36 from 1660053 to 2293157 tid 2 2014-08-08T10:52:37.953 INFO:tasks.rados.rados.0.vpm137.stdout:3951: writing vpm1375878-36 from 2992226 to 3211446 tid 3 2014-08-08T10:52:37.955 INFO:tasks.rados.rados.0.vpm137.stdout:3941: expect (ObjNum 2374 snap 0 seq_num 2374) 2014-08-08T10:52:38.542 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:38.542 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:38.678 INFO:tasks.rados.rados.0.vpm137.stdout:3941: done (9 left) 2014-08-08T10:52:38.678 INFO:tasks.rados.rados.0.vpm137.stdout:3952: write oid 309 current snap is 0 2014-08-08T10:52:38.679 INFO:tasks.rados.rados.0.vpm137.stdout:3952: seq_num 2411 ranges {507700=741455,1774185=454723,2696299=211401} 2014-08-08T10:52:38.719 INFO:tasks.rados.rados.0.vpm137.stdout:3952: writing vpm1375878-309 from 507700 to 1249155 tid 1 2014-08-08T10:52:38.753 INFO:tasks.rados.rados.0.vpm137.stdout:3952: writing vpm1375878-309 from 1774185 to 2228908 tid 2 2014-08-08T10:52:38.767 INFO:tasks.rados.rados.0.vpm137.stdout:3952: writing vpm1375878-309 from 2696299 to 2907700 tid 3 2014-08-08T10:52:38.768 INFO:tasks.rados.rados.0.vpm137.stdout:3944: finishing write tid 1 to vpm1375878-74 2014-08-08T10:52:38.769 INFO:tasks.rados.rados.0.vpm137.stdout:3944: finishing write tid 2 to vpm1375878-74 2014-08-08T10:52:38.769 INFO:tasks.rados.rados.0.vpm137.stdout:3944: finishing write tid 4 to vpm1375878-74 2014-08-08T10:52:38.769 INFO:tasks.rados.rados.0.vpm137.stdout:3944: finishing write tid 5 to vpm1375878-74 2014-08-08T10:52:38.771 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 74 v 465 (ObjNum 2407 snap 0 seq_num 2407) dirty exists 2014-08-08T10:52:38.771 INFO:tasks.rados.rados.0.vpm137.stdout:3945: expect (ObjNum 993 snap 0 seq_num 993) 2014-08-08T10:52:38.802 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:39.321 INFO:tasks.rados.rados.0.vpm137.stdout:3943: done (9 left) 2014-08-08T10:52:39.322 INFO:tasks.rados.rados.0.vpm137.stdout:3944: done (8 left) 2014-08-08T10:52:39.322 INFO:tasks.rados.rados.0.vpm137.stdout:3945: done (7 left) 2014-08-08T10:52:39.322 INFO:tasks.rados.rados.0.vpm137.stdout:3953: write oid 392 current snap is 0 2014-08-08T10:52:39.323 INFO:tasks.rados.rados.0.vpm137.stdout:3953: seq_num 2412 ranges {416434=714283,1216433=1} 2014-08-08T10:52:39.347 INFO:tasks.rados.rados.0.vpm137.stdout:3953: writing vpm1375878-392 from 416434 to 1130717 tid 1 2014-08-08T10:52:39.355 INFO:tasks.rados.rados.0.vpm137.stdout:3953: writing vpm1375878-392 from 1216433 to 1216434 tid 2 2014-08-08T10:52:39.356 INFO:tasks.rados.rados.0.vpm137.stdout:3946: expect (ObjNum 2138 snap 0 seq_num 2138) 2014-08-08T10:52:40.969 INFO:tasks.rados.rados.0.vpm137.stdout:3946: done (7 left) 2014-08-08T10:52:40.969 INFO:tasks.rados.rados.0.vpm137.stdout:3954: write oid 337 current snap is 0 2014-08-08T10:52:40.969 INFO:tasks.rados.rados.0.vpm137.stdout:3954: seq_num 2413 ranges {511663=400000} 2014-08-08T10:52:40.986 INFO:tasks.rados.rados.0.vpm137.stdout:3954: writing vpm1375878-337 from 511663 to 911663 tid 1 2014-08-08T10:52:40.988 INFO:tasks.rados.rados.0.vpm137.stdout:3947: finishing write tid 1 to vpm1375878-433 2014-08-08T10:52:40.989 INFO:tasks.rados.rados.0.vpm137.stdout:3947: finishing write tid 2 to vpm1375878-433 2014-08-08T10:52:40.990 INFO:tasks.rados.rados.0.vpm137.stdout:3947: finishing write tid 4 to vpm1375878-433 2014-08-08T10:52:40.990 INFO:tasks.rados.rados.0.vpm137.stdout:3947: finishing write tid 5 to vpm1375878-433 2014-08-08T10:52:40.991 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 433 v 468 (ObjNum 2408 snap 0 seq_num 2408) dirty exists 2014-08-08T10:52:40.991 INFO:tasks.rados.rados.0.vpm137.stdout:3948: expect (ObjNum 2164 snap 0 seq_num 2164) 2014-08-08T10:52:41.836 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:43.102 INFO:tasks.rados.rados.0.vpm137.stdout:3947: done (7 left) 2014-08-08T10:52:43.102 INFO:tasks.rados.rados.0.vpm137.stdout:3948: done (6 left) 2014-08-08T10:52:43.103 INFO:tasks.rados.rados.0.vpm137.stdout:3955: read oid 255 snap -1 2014-08-08T10:52:43.104 INFO:tasks.rados.rados.0.vpm137.stdout:3956: write oid 334 current snap is 0 2014-08-08T10:52:43.104 INFO:tasks.rados.rados.0.vpm137.stdout:3956: seq_num 2414 ranges {607916=1} 2014-08-08T10:52:43.108 INFO:tasks.rados.rados.0.vpm137.stdout:3956: writing vpm1375878-334 from 607916 to 607917 tid 1 2014-08-08T10:52:43.110 INFO:tasks.rados.rados.0.vpm137.stdout:3949: finishing write tid 1 to vpm1375878-160 2014-08-08T10:52:43.110 INFO:tasks.rados.rados.0.vpm137.stdout:3949: finishing write tid 2 to vpm1375878-160 2014-08-08T10:52:43.110 INFO:tasks.rados.rados.0.vpm137.stdout:3949: finishing write tid 3 to vpm1375878-160 2014-08-08T10:52:43.110 INFO:tasks.rados.rados.0.vpm137.stdout:3949: finishing write tid 5 to vpm1375878-160 2014-08-08T10:52:43.111 INFO:tasks.rados.rados.0.vpm137.stdout:3949: finishing write tid 6 to vpm1375878-160 2014-08-08T10:52:43.111 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 160 v 416 (ObjNum 2409 snap 0 seq_num 2409) dirty exists 2014-08-08T10:52:43.111 INFO:tasks.rados.rados.0.vpm137.stdout:3950: expect (ObjNum 2300 snap 0 seq_num 2300) 2014-08-08T10:52:43.208 INFO:tasks.rados.rados.0.vpm137.stdout:3949: done (7 left) 2014-08-08T10:52:43.208 INFO:tasks.rados.rados.0.vpm137.stdout:3950: done (6 left) 2014-08-08T10:52:43.209 INFO:tasks.rados.rados.0.vpm137.stdout:3957: read oid 182 snap -1 2014-08-08T10:52:43.209 INFO:tasks.rados.rados.0.vpm137.stdout:3958: delete oid 294 current snap is 0 2014-08-08T10:52:43.209 INFO:tasks.rados.rados.0.vpm137.stdout:3951: finishing write tid 1 to vpm1375878-36 2014-08-08T10:52:43.209 INFO:tasks.rados.rados.0.vpm137.stdout:3951: finishing write tid 2 to vpm1375878-36 2014-08-08T10:52:43.210 INFO:tasks.rados.rados.0.vpm137.stdout:3951: finishing write tid 3 to vpm1375878-36 2014-08-08T10:52:43.210 INFO:tasks.rados.rados.0.vpm137.stdout:3951: finishing write tid 5 to vpm1375878-36 2014-08-08T10:52:43.210 INFO:tasks.rados.rados.0.vpm137.stdout:3951: finishing write tid 6 to vpm1375878-36 2014-08-08T10:52:43.211 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 36 v 453 (ObjNum 2410 snap 0 seq_num 2410) dirty exists 2014-08-08T10:52:43.220 INFO:tasks.rados.rados.0.vpm137.stdout:3952: finishing write tid 1 to vpm1375878-309 2014-08-08T10:52:43.221 INFO:tasks.rados.rados.0.vpm137.stdout:3952: finishing write tid 2 to vpm1375878-309 2014-08-08T10:52:43.221 INFO:tasks.rados.rados.0.vpm137.stdout:3952: finishing write tid 3 to vpm1375878-309 2014-08-08T10:52:43.221 INFO:tasks.rados.rados.0.vpm137.stdout:3952: finishing write tid 5 to vpm1375878-309 2014-08-08T10:52:43.222 INFO:tasks.rados.rados.0.vpm137.stdout:3952: finishing write tid 6 to vpm1375878-309 2014-08-08T10:52:43.222 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 309 v 472 (ObjNum 2411 snap 0 seq_num 2411) dirty exists 2014-08-08T10:52:43.222 INFO:tasks.rados.rados.0.vpm137.stdout:3953: finishing write tid 1 to vpm1375878-392 2014-08-08T10:52:43.222 INFO:tasks.rados.rados.0.vpm137.stdout:3953: finishing write tid 2 to vpm1375878-392 2014-08-08T10:52:43.223 INFO:tasks.rados.rados.0.vpm137.stdout:3953: finishing write tid 4 to vpm1375878-392 2014-08-08T10:52:43.223 INFO:tasks.rados.rados.0.vpm137.stdout:3953: finishing write tid 5 to vpm1375878-392 2014-08-08T10:52:43.223 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 392 v 438 (ObjNum 2412 snap 0 seq_num 2412) dirty exists 2014-08-08T10:52:43.224 INFO:tasks.rados.rados.0.vpm137.stdout:3954: finishing write tid 1 to vpm1375878-337 2014-08-08T10:52:43.224 INFO:tasks.rados.rados.0.vpm137.stdout:3954: finishing write tid 3 to vpm1375878-337 2014-08-08T10:52:43.224 INFO:tasks.rados.rados.0.vpm137.stdout:3954: finishing write tid 4 to vpm1375878-337 2014-08-08T10:52:43.225 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 337 v 326 (ObjNum 2413 snap 0 seq_num 2413) dirty exists 2014-08-08T10:52:43.271 INFO:tasks.rados.rados.0.vpm137.stdout:3955: expect (ObjNum 1989 snap 0 seq_num 1989) 2014-08-08T10:52:43.561 INFO:tasks.rados.rados.0.vpm137.stdout:3951: done (7 left) 2014-08-08T10:52:43.562 INFO:tasks.rados.rados.0.vpm137.stdout:3952: done (6 left) 2014-08-08T10:52:43.562 INFO:tasks.rados.rados.0.vpm137.stdout:3953: done (5 left) 2014-08-08T10:52:43.563 INFO:tasks.rados.rados.0.vpm137.stdout:3954: done (4 left) 2014-08-08T10:52:43.563 INFO:tasks.rados.rados.0.vpm137.stdout:3955: done (3 left) 2014-08-08T10:52:43.563 INFO:tasks.rados.rados.0.vpm137.stdout:3957: done (2 left) 2014-08-08T10:52:43.563 INFO:tasks.rados.rados.0.vpm137.stdout:3958: done (1 left) 2014-08-08T10:52:43.563 INFO:tasks.rados.rados.0.vpm137.stdout:3959: write oid 190 current snap is 0 2014-08-08T10:52:43.564 INFO:tasks.rados.rados.0.vpm137.stdout:3959: seq_num 2416 ranges {496553=637597,1569806=439747,2096552=1} 2014-08-08T10:52:43.587 INFO:tasks.rados.rados.0.vpm137.stdout:3959: writing vpm1375878-190 from 496553 to 1134150 tid 1 2014-08-08T10:52:43.614 INFO:tasks.rados.rados.0.vpm137.stdout:3959: writing vpm1375878-190 from 1569806 to 2009553 tid 2 2014-08-08T10:52:43.617 INFO:tasks.rados.rados.0.vpm137.stdout:3959: writing vpm1375878-190 from 2096552 to 2096553 tid 3 2014-08-08T10:52:43.618 INFO:tasks.rados.rados.0.vpm137.stdout:3956: finishing write tid 1 to vpm1375878-334 2014-08-08T10:52:43.618 INFO:tasks.rados.rados.0.vpm137.stdout:3956: finishing write tid 3 to vpm1375878-334 2014-08-08T10:52:43.619 INFO:tasks.rados.rados.0.vpm137.stdout:3956: finishing write tid 4 to vpm1375878-334 2014-08-08T10:52:43.620 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 334 v 647 (ObjNum 2414 snap 0 seq_num 2414) dirty exists 2014-08-08T10:52:43.620 INFO:tasks.rados.rados.0.vpm137.stdout:3959: finishing write tid 1 to vpm1375878-190 2014-08-08T10:52:43.621 INFO:tasks.rados.rados.0.vpm137.stdout:3956: done (1 left) 2014-08-08T10:52:43.621 INFO:tasks.rados.rados.0.vpm137.stdout:3960: write oid 413 current snap is 0 2014-08-08T10:52:43.624 INFO:tasks.rados.rados.0.vpm137.stdout:3960: seq_num 2417 ranges {708134=760825,1915617=392517} 2014-08-08T10:52:43.668 INFO:tasks.rados.rados.0.vpm137.stdout:3960: writing vpm1375878-413 from 708134 to 1468959 tid 1 2014-08-08T10:52:43.690 INFO:tasks.rados.rados.0.vpm137.stdout:3960: writing vpm1375878-413 from 1915617 to 2308134 tid 2 2014-08-08T10:52:43.694 INFO:tasks.rados.rados.0.vpm137.stdout:3959: finishing write tid 2 to vpm1375878-190 2014-08-08T10:52:43.694 INFO:tasks.rados.rados.0.vpm137.stdout:3959: finishing write tid 3 to vpm1375878-190 2014-08-08T10:52:43.694 INFO:tasks.rados.rados.0.vpm137.stdout:3959: finishing write tid 5 to vpm1375878-190 2014-08-08T10:52:43.694 INFO:tasks.rados.rados.0.vpm137.stdout:3959: finishing write tid 6 to vpm1375878-190 2014-08-08T10:52:43.695 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 190 v 476 (ObjNum 2416 snap 0 seq_num 2416) dirty exists 2014-08-08T10:52:43.695 INFO:tasks.rados.rados.0.vpm137.stdout:3959: done (1 left) 2014-08-08T10:52:43.695 INFO:tasks.rados.rados.0.vpm137.stdout:3961: read oid 162 snap -1 2014-08-08T10:52:43.696 INFO:tasks.rados.rados.0.vpm137.stdout:3962: write oid 261 current snap is 0 2014-08-08T10:52:43.696 INFO:tasks.rados.rados.0.vpm137.stdout:3962: seq_num 2418 ranges {652964=616278,1761974=731561,3052963=1} 2014-08-08T10:52:43.718 INFO:tasks.rados.rados.0.vpm137.stdout:3962: writing vpm1375878-261 from 652964 to 1269242 tid 1 2014-08-08T10:52:43.773 INFO:tasks.rados.rados.0.vpm137.stdout:3962: writing vpm1375878-261 from 1761974 to 2493535 tid 2 2014-08-08T10:52:43.781 INFO:tasks.rados.rados.0.vpm137.stdout:3962: writing vpm1375878-261 from 3052963 to 3052964 tid 3 2014-08-08T10:52:43.781 INFO:tasks.rados.rados.0.vpm137.stdout:3960: finishing write tid 1 to vpm1375878-413 2014-08-08T10:52:43.781 INFO:tasks.rados.rados.0.vpm137.stdout:3960: finishing write tid 2 to vpm1375878-413 2014-08-08T10:52:43.782 INFO:tasks.rados.rados.0.vpm137.stdout:3961: expect (ObjNum 972 snap 0 seq_num 972) 2014-08-08T10:52:45.048 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:45.115 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:45.344 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 853383851, u'bytes_used': 1700204544, u'op_per_sec': 7, u'version': 1704, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 1396393, u'write_bytes_sec': 680488, u'bytes_avail': 427586809856}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:45.345 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:45.544 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:45.545 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:45.641 INFO:tasks.rados.rados.0.vpm137.stdout:3961: done (2 left) 2014-08-08T10:52:45.642 INFO:tasks.rados.rados.0.vpm137.stdout:3963: read oid 248 snap -1 2014-08-08T10:52:45.642 INFO:tasks.rados.rados.0.vpm137.stdout:3964: delete oid 334 current snap is 0 2014-08-08T10:52:45.644 INFO:tasks.rados.rados.0.vpm137.stdout:3960: finishing write tid 4 to vpm1375878-413 2014-08-08T10:52:45.644 INFO:tasks.rados.rados.0.vpm137.stdout:3960: finishing write tid 5 to vpm1375878-413 2014-08-08T10:52:45.645 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 413 v 522 (ObjNum 2417 snap 0 seq_num 2417) dirty exists 2014-08-08T10:52:45.645 INFO:tasks.rados.rados.0.vpm137.stdout:3962: finishing write tid 1 to vpm1375878-261 2014-08-08T10:52:45.645 INFO:tasks.rados.rados.0.vpm137.stdout:3962: finishing write tid 2 to vpm1375878-261 2014-08-08T10:52:45.646 INFO:tasks.rados.rados.0.vpm137.stdout:3962: finishing write tid 3 to vpm1375878-261 2014-08-08T10:52:45.646 INFO:tasks.rados.rados.0.vpm137.stdout:3962: finishing write tid 5 to vpm1375878-261 2014-08-08T10:52:45.646 INFO:tasks.rados.rados.0.vpm137.stdout:3962: finishing write tid 6 to vpm1375878-261 2014-08-08T10:52:45.646 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 261 v 457 (ObjNum 2418 snap 0 seq_num 2418) dirty exists 2014-08-08T10:52:45.751 INFO:tasks.rados.rados.0.vpm137.stdout:3960: done (3 left) 2014-08-08T10:52:45.752 INFO:tasks.rados.rados.0.vpm137.stdout:3962: done (2 left) 2014-08-08T10:52:45.752 INFO:tasks.rados.rados.0.vpm137.stdout:3964: done (1 left) 2014-08-08T10:52:45.752 INFO:tasks.rados.rados.0.vpm137.stdout:3965: delete oid 101 current snap is 0 2014-08-08T10:52:45.778 INFO:tasks.rados.rados.0.vpm137.stdout:3963: expect (ObjNum 1687 snap 0 seq_num 1687) 2014-08-08T10:52:45.809 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:46.677 INFO:tasks.rados.rados.0.vpm137.stdout:3963: done (1 left) 2014-08-08T10:52:46.677 INFO:tasks.rados.rados.0.vpm137.stdout:3965: done (0 left) 2014-08-08T10:52:46.678 INFO:tasks.rados.rados.0.vpm137.stdout:3966: read oid 268 snap -1 2014-08-08T10:52:46.679 INFO:tasks.rados.rados.0.vpm137.stdout:3967: write oid 323 current snap is 0 2014-08-08T10:52:46.679 INFO:tasks.rados.rados.0.vpm137.stdout:3967: seq_num 2421 ranges {753180=1} 2014-08-08T10:52:46.703 INFO:tasks.rados.rados.0.vpm137.stdout:3967: writing vpm1375878-323 from 753180 to 753181 tid 1 2014-08-08T10:52:46.705 INFO:tasks.rados.rados.0.vpm137.stdout:3966: expect (ObjNum 1630 snap 0 seq_num 1630) 2014-08-08T10:52:47.390 INFO:tasks.rados.rados.0.vpm137.stdout:3966: done (1 left) 2014-08-08T10:52:47.391 INFO:tasks.rados.rados.0.vpm137.stdout:3968: write oid 401 current snap is 0 2014-08-08T10:52:47.391 INFO:tasks.rados.rados.0.vpm137.stdout:3968: seq_num 2422 ranges {696987=692039,1496986=1} 2014-08-08T10:52:47.436 INFO:tasks.rados.rados.0.vpm137.stdout:3968: writing vpm1375878-401 from 696987 to 1389026 tid 1 2014-08-08T10:52:47.447 INFO:tasks.rados.rados.0.vpm137.stdout:3968: writing vpm1375878-401 from 1496986 to 1496987 tid 2 2014-08-08T10:52:47.448 INFO:tasks.rados.rados.0.vpm137.stdout:3967: finishing write tid 1 to vpm1375878-323 2014-08-08T10:52:47.449 INFO:tasks.rados.rados.0.vpm137.stdout:3967: finishing write tid 3 to vpm1375878-323 2014-08-08T10:52:47.450 INFO:tasks.rados.rados.0.vpm137.stdout:3967: finishing write tid 4 to vpm1375878-323 2014-08-08T10:52:47.450 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 323 v 379 (ObjNum 2421 snap 0 seq_num 2421) dirty exists 2014-08-08T10:52:47.450 INFO:tasks.rados.rados.0.vpm137.stdout:3967: done (1 left) 2014-08-08T10:52:47.451 INFO:tasks.rados.rados.0.vpm137.stdout:3969: write oid 83 current snap is 0 2014-08-08T10:52:47.451 INFO:tasks.rados.rados.0.vpm137.stdout:3969: seq_num 2423 ranges {793241=400000} 2014-08-08T10:52:47.481 INFO:tasks.rados.rados.0.vpm137.stdout:3969: writing vpm1375878-83 from 793241 to 1193241 tid 1 2014-08-08T10:52:47.483 INFO:tasks.rados.rados.0.vpm137.stdout:3968: finishing write tid 1 to vpm1375878-401 2014-08-08T10:52:47.483 INFO:tasks.rados.rados.0.vpm137.stdout:3970: read oid 172 snap -1 2014-08-08T10:52:47.484 INFO:tasks.rados.rados.0.vpm137.stdout:3971: delete oid 335 current snap is 0 2014-08-08T10:52:47.552 INFO:tasks.rados.rados.0.vpm137.stdout:3970: expect (ObjNum 921 snap 0 seq_num 921) 2014-08-08T10:52:48.797 INFO:tasks.rados.rados.0.vpm137.stdout:3970: done (3 left) 2014-08-08T10:52:48.797 INFO:tasks.rados.rados.0.vpm137.stdout:3971: done (2 left) 2014-08-08T10:52:48.798 INFO:tasks.rados.rados.0.vpm137.stdout:3972: write oid 219 current snap is 0 2014-08-08T10:52:48.798 INFO:tasks.rados.rados.0.vpm137.stdout:3972: seq_num 2425 ranges {681876=510300,1652611=609424,2681875=1} 2014-08-08T10:52:48.818 INFO:tasks.rados.rados.0.vpm137.stdout:3972: writing vpm1375878-219 from 681876 to 1192176 tid 1 2014-08-08T10:52:48.842 INFO:tasks.rados.rados.0.vpm137.stdout:3972: writing vpm1375878-219 from 1652611 to 2262035 tid 2 2014-08-08T10:52:48.846 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:48.848 INFO:tasks.rados.rados.0.vpm137.stdout:3972: writing vpm1375878-219 from 2681875 to 2681876 tid 3 2014-08-08T10:52:48.851 INFO:tasks.rados.rados.0.vpm137.stdout:3968: finishing write tid 2 to vpm1375878-401 2014-08-08T10:52:48.851 INFO:tasks.rados.rados.0.vpm137.stdout:3969: finishing write tid 1 to vpm1375878-83 2014-08-08T10:52:48.852 INFO:tasks.rados.rados.0.vpm137.stdout:3969: finishing write tid 3 to vpm1375878-83 2014-08-08T10:52:48.853 INFO:tasks.rados.rados.0.vpm137.stdout:3968: finishing write tid 4 to vpm1375878-401 2014-08-08T10:52:48.853 INFO:tasks.rados.rados.0.vpm137.stdout:3969: finishing write tid 4 to vpm1375878-83 2014-08-08T10:52:48.854 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 83 v 328 (ObjNum 2423 snap 0 seq_num 2423) dirty exists 2014-08-08T10:52:48.854 INFO:tasks.rados.rados.0.vpm137.stdout:3968: finishing write tid 5 to vpm1375878-401 2014-08-08T10:52:48.855 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 401 v 651 (ObjNum 2422 snap 0 seq_num 2422) dirty exists 2014-08-08T10:52:48.871 INFO:tasks.rados.rados.0.vpm137.stdout:3968: done (2 left) 2014-08-08T10:52:48.871 INFO:tasks.rados.rados.0.vpm137.stdout:3969: done (1 left) 2014-08-08T10:52:48.871 INFO:tasks.rados.rados.0.vpm137.stdout:3973: read oid 348 snap -1 2014-08-08T10:52:48.871 INFO:tasks.rados.rados.0.vpm137.stdout:3974: read oid 314 snap -1 2014-08-08T10:52:48.872 INFO:tasks.rados.rados.0.vpm137.stdout:3975: read oid 51 snap -1 2014-08-08T10:52:48.872 INFO:tasks.rados.rados.0.vpm137.stdout:3976: write oid 496 current snap is 0 2014-08-08T10:52:48.872 INFO:tasks.rados.rados.0.vpm137.stdout:3976: seq_num 2426 ranges {494482=766776,2016343=679619,2894481=1} 2014-08-08T10:52:48.878 INFO:tasks.rados.rados.0.vpm137.stdout:3976: writing vpm1375878-496 from 494482 to 1261258 tid 1 2014-08-08T10:52:48.914 INFO:tasks.rados.rados.0.vpm137.stdout:3976: writing vpm1375878-496 from 2016343 to 2695962 tid 2 2014-08-08T10:52:48.922 INFO:tasks.rados.rados.0.vpm137.stdout:3976: writing vpm1375878-496 from 2894481 to 2894482 tid 3 2014-08-08T10:52:48.922 INFO:tasks.rados.rados.0.vpm137.stdout:3972: finishing write tid 1 to vpm1375878-219 2014-08-08T10:52:48.925 INFO:tasks.rados.rados.0.vpm137.stdout:3972: finishing write tid 2 to vpm1375878-219 2014-08-08T10:52:48.925 INFO:tasks.rados.rados.0.vpm137.stdout:3972: finishing write tid 3 to vpm1375878-219 2014-08-08T10:52:48.926 INFO:tasks.rados.rados.0.vpm137.stdout:3972: finishing write tid 5 to vpm1375878-219 2014-08-08T10:52:48.926 INFO:tasks.rados.rados.0.vpm137.stdout:3972: finishing write tid 6 to vpm1375878-219 2014-08-08T10:52:48.927 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 219 v 454 (ObjNum 2425 snap 0 seq_num 2425) dirty exists 2014-08-08T10:52:48.927 INFO:tasks.rados.rados.0.vpm137.stdout:3974: expect (ObjNum 1744 snap 0 seq_num 1744) 2014-08-08T10:52:49.482 INFO:tasks.rados.rados.0.vpm137.stdout:3972: done (4 left) 2014-08-08T10:52:49.482 INFO:tasks.rados.rados.0.vpm137.stdout:3974: done (3 left) 2014-08-08T10:52:49.483 INFO:tasks.rados.rados.0.vpm137.stdout:3977: delete oid 94 current snap is 0 2014-08-08T10:52:49.483 INFO:tasks.rados.rados.0.vpm137.stdout:3975: expect (ObjNum 2365 snap 0 seq_num 2365) 2014-08-08T10:52:49.510 INFO:tasks.rados.rados.0.vpm137.stdout:3975: done (3 left) 2014-08-08T10:52:49.510 INFO:tasks.rados.rados.0.vpm137.stdout:3977: done (2 left) 2014-08-08T10:52:49.510 INFO:tasks.rados.rados.0.vpm137.stdout:3978: write oid 442 current snap is 0 2014-08-08T10:52:49.510 INFO:tasks.rados.rados.0.vpm137.stdout:3978: seq_num 2428 ranges {383116=1} 2014-08-08T10:52:49.513 INFO:tasks.rados.rados.0.vpm137.stdout:3978: writing vpm1375878-442 from 383116 to 383117 tid 1 2014-08-08T10:52:49.513 INFO:tasks.rados.rados.0.vpm137.stdout:3979: write oid 237 current snap is 0 2014-08-08T10:52:49.513 INFO:tasks.rados.rados.0.vpm137.stdout:3979: seq_num 2429 ranges {78346=1} 2014-08-08T10:52:49.514 INFO:tasks.rados.rados.0.vpm137.stdout:3979: writing vpm1375878-237 from 78346 to 78347 tid 1 2014-08-08T10:52:49.515 INFO:tasks.rados.rados.0.vpm137.stdout:3973: expect (ObjNum 1622 snap 0 seq_num 1622) 2014-08-08T10:52:50.311 INFO:tasks.rados.rados.0.vpm137.stdout:3973: done (3 left) 2014-08-08T10:52:50.312 INFO:tasks.rados.rados.0.vpm137.stdout:3980: read oid 45 snap -1 2014-08-08T10:52:50.312 INFO:tasks.rados.rados.0.vpm137.stdout:3981: read oid 168 snap -1 2014-08-08T10:52:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3982: read oid 17 snap -1 2014-08-08T10:52:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3983: read oid 305 snap -1 2014-08-08T10:52:50.313 INFO:tasks.rados.rados.0.vpm137.stdout:3984: read oid 24 snap -1 2014-08-08T10:52:50.314 INFO:tasks.rados.rados.0.vpm137.stdout:3985: write oid 69 current snap is 0 2014-08-08T10:52:50.314 INFO:tasks.rados.rados.0.vpm137.stdout:3985: seq_num 2430 ranges {538505=407466,1338504=1} 2014-08-08T10:52:50.341 INFO:tasks.rados.rados.0.vpm137.stdout:3985: writing vpm1375878-69 from 538505 to 945971 tid 1 2014-08-08T10:52:50.347 INFO:tasks.rados.rados.0.vpm137.stdout:3985: writing vpm1375878-69 from 1338504 to 1338505 tid 2 2014-08-08T10:52:50.348 INFO:tasks.rados.rados.0.vpm137.stdout:3976: finishing write tid 1 to vpm1375878-496 2014-08-08T10:52:50.348 INFO:tasks.rados.rados.0.vpm137.stdout:3976: finishing write tid 2 to vpm1375878-496 2014-08-08T10:52:50.349 INFO:tasks.rados.rados.0.vpm137.stdout:3976: finishing write tid 3 to vpm1375878-496 2014-08-08T10:52:50.349 INFO:tasks.rados.rados.0.vpm137.stdout:3976: finishing write tid 5 to vpm1375878-496 2014-08-08T10:52:50.350 INFO:tasks.rados.rados.0.vpm137.stdout:3976: finishing write tid 6 to vpm1375878-496 2014-08-08T10:52:50.350 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 496 v 420 (ObjNum 2426 snap 0 seq_num 2426) dirty exists 2014-08-08T10:52:50.350 INFO:tasks.rados.rados.0.vpm137.stdout:3978: finishing write tid 1 to vpm1375878-442 2014-08-08T10:52:50.350 INFO:tasks.rados.rados.0.vpm137.stdout:3978: finishing write tid 3 to vpm1375878-442 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:3979: finishing write tid 1 to vpm1375878-237 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:3978: finishing write tid 4 to vpm1375878-442 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 442 v 330 (ObjNum 2428 snap 0 seq_num 2428) dirty exists 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:3979: finishing write tid 3 to vpm1375878-237 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:3979: finishing write tid 4 to vpm1375878-237 2014-08-08T10:52:50.351 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 237 v 653 (ObjNum 2429 snap 0 seq_num 2429) dirty exists 2014-08-08T10:52:50.352 INFO:tasks.rados.rados.0.vpm137.stdout:3980: expect (ObjNum 665 snap 0 seq_num 665) 2014-08-08T10:52:50.400 INFO:tasks.rados.rados.0.vpm137.stdout:3976: done (8 left) 2014-08-08T10:52:50.400 INFO:tasks.rados.rados.0.vpm137.stdout:3978: done (7 left) 2014-08-08T10:52:50.400 INFO:tasks.rados.rados.0.vpm137.stdout:3979: done (6 left) 2014-08-08T10:52:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3980: done (5 left) 2014-08-08T10:52:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3981: done (4 left) 2014-08-08T10:52:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3986: write oid 360 current snap is 0 2014-08-08T10:52:50.401 INFO:tasks.rados.rados.0.vpm137.stdout:3986: seq_num 2431 ranges {634759=400000} 2014-08-08T10:52:50.420 INFO:tasks.rados.rados.0.vpm137.stdout:3986: writing vpm1375878-360 from 634759 to 1034759 tid 1 2014-08-08T10:52:50.423 INFO:tasks.rados.rados.0.vpm137.stdout:3982: expect (ObjNum 2304 snap 0 seq_num 2304) 2014-08-08T10:52:50.450 INFO:tasks.rados.rados.0.vpm137.stdout:3982: done (4 left) 2014-08-08T10:52:50.451 INFO:tasks.rados.rados.0.vpm137.stdout:3987: read oid 218 snap -1 2014-08-08T10:52:50.451 INFO:tasks.rados.rados.0.vpm137.stdout:3988: read oid 186 snap -1 2014-08-08T10:52:50.451 INFO:tasks.rados.rados.0.vpm137.stdout:3989: read oid 485 snap -1 2014-08-08T10:52:50.451 INFO:tasks.rados.rados.0.vpm137.stdout:3990: delete oid 363 current snap is 0 2014-08-08T10:52:50.452 INFO:tasks.rados.rados.0.vpm137.stdout:3983: expect (ObjNum 1326 snap 0 seq_num 1326) 2014-08-08T10:52:50.717 INFO:tasks.rados.rados.0.vpm137.stdout:3983: done (7 left) 2014-08-08T10:52:50.717 INFO:tasks.rados.rados.0.vpm137.stdout:3990: done (6 left) 2014-08-08T10:52:50.717 INFO:tasks.rados.rados.0.vpm137.stdout:3991: write oid 135 current snap is 0 2014-08-08T10:52:50.718 INFO:tasks.rados.rados.0.vpm137.stdout:3991: seq_num 2433 ranges {523394=510399,1756580=707881,2523393=1} 2014-08-08T10:52:50.737 INFO:tasks.rados.rados.0.vpm137.stdout:3991: writing vpm1375878-135 from 523394 to 1033793 tid 1 2014-08-08T10:52:50.766 INFO:tasks.rados.rados.0.vpm137.stdout:3991: writing vpm1375878-135 from 1756580 to 2464461 tid 2 2014-08-08T10:52:50.769 INFO:tasks.rados.rados.0.vpm137.stdout:3991: writing vpm1375878-135 from 2523393 to 2523394 tid 3 2014-08-08T10:52:50.770 INFO:tasks.rados.rados.0.vpm137.stdout:3984: expect (ObjNum 2272 snap 0 seq_num 2272) 2014-08-08T10:52:52.272 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:52.326 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:52.639 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 839380566, u'bytes_used': 1693048832, u'op_per_sec': 15, u'version': 1706, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2485439, u'write_bytes_sec': 845641, u'bytes_avail': 427593965568}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:52.641 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:52.844 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:52.845 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:53.070 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:53.605 INFO:tasks.rados.rados.0.vpm137.stdout:3984: done (6 left) 2014-08-08T10:52:53.606 INFO:tasks.rados.rados.0.vpm137.stdout:3992: write oid 269 current snap is 0 2014-08-08T10:52:53.606 INFO:tasks.rados.rados.0.vpm137.stdout:3992: seq_num 2434 ranges {619648=482203,1870712=348936} 2014-08-08T10:52:53.639 INFO:tasks.rados.rados.0.vpm137.stdout:3992: writing vpm1375878-269 from 619648 to 1101851 tid 1 2014-08-08T10:52:53.670 INFO:tasks.rados.rados.0.vpm137.stdout:3992: writing vpm1375878-269 from 1870712 to 2219648 tid 2 2014-08-08T10:52:53.673 INFO:tasks.rados.rados.0.vpm137.stdout:3985: finishing write tid 1 to vpm1375878-69 2014-08-08T10:52:53.674 INFO:tasks.rados.rados.0.vpm137.stdout:3985: finishing write tid 2 to vpm1375878-69 2014-08-08T10:52:53.674 INFO:tasks.rados.rados.0.vpm137.stdout:3985: finishing write tid 4 to vpm1375878-69 2014-08-08T10:52:53.674 INFO:tasks.rados.rados.0.vpm137.stdout:3985: finishing write tid 5 to vpm1375878-69 2014-08-08T10:52:53.674 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 69 v 382 (ObjNum 2430 snap 0 seq_num 2430) dirty exists 2014-08-08T10:52:53.675 INFO:tasks.rados.rados.0.vpm137.stdout:3986: finishing write tid 1 to vpm1375878-360 2014-08-08T10:52:53.675 INFO:tasks.rados.rados.0.vpm137.stdout:3986: finishing write tid 3 to vpm1375878-360 2014-08-08T10:52:53.675 INFO:tasks.rados.rados.0.vpm137.stdout:3986: finishing write tid 4 to vpm1375878-360 2014-08-08T10:52:53.676 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 360 v 435 (ObjNum 2431 snap 0 seq_num 2431) dirty exists 2014-08-08T10:52:53.676 INFO:tasks.rados.rados.0.vpm137.stdout:3988: expect (ObjNum 2329 snap 0 seq_num 2329) 2014-08-08T10:52:55.204 INFO:tasks.rados.rados.0.vpm137.stdout:3985: done (6 left) 2014-08-08T10:52:55.204 INFO:tasks.rados.rados.0.vpm137.stdout:3986: done (5 left) 2014-08-08T10:52:55.205 INFO:tasks.rados.rados.0.vpm137.stdout:3987: done (4 left) 2014-08-08T10:52:55.205 INFO:tasks.rados.rados.0.vpm137.stdout:3988: done (3 left) 2014-08-08T10:52:55.206 INFO:tasks.rados.rados.0.vpm137.stdout:3993: read oid 450 snap -1 2014-08-08T10:52:55.206 INFO:tasks.rados.rados.0.vpm137.stdout:3994: read oid 476 snap -1 2014-08-08T10:52:55.206 INFO:tasks.rados.rados.0.vpm137.stdout:3995: read oid 394 snap -1 2014-08-08T10:52:55.207 INFO:tasks.rados.rados.0.vpm137.stdout:3996: write oid 160 current snap is 0 2014-08-08T10:52:55.207 INFO:tasks.rados.rados.0.vpm137.stdout:3996: seq_num 2435 ranges {679806=489079,1948748=599696,3342668=137138} 2014-08-08T10:52:55.269 INFO:tasks.rados.rados.0.vpm137.stdout:3996: writing vpm1375878-160 from 679806 to 1168885 tid 1 2014-08-08T10:52:55.301 INFO:tasks.rados.rados.0.vpm137.stdout:3996: writing vpm1375878-160 from 1948748 to 2548444 tid 2 2014-08-08T10:52:55.313 INFO:tasks.rados.rados.0.vpm137.stdout:3996: writing vpm1375878-160 from 3342668 to 3479806 tid 3 2014-08-08T10:52:55.315 INFO:tasks.rados.rados.0.vpm137.stdout:3989: expect (ObjNum 2189 snap 0 seq_num 2189) 2014-08-08T10:52:56.116 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:56.312 INFO:tasks.rados.rados.0.vpm137.stdout:3989: done (6 left) 2014-08-08T10:52:56.313 INFO:tasks.rados.rados.0.vpm137.stdout:3997: read oid 441 snap -1 2014-08-08T10:52:56.313 INFO:tasks.rados.rados.0.vpm137.stdout:3998: read oid 308 snap -1 2014-08-08T10:52:56.314 INFO:tasks.rados.rados.0.vpm137.stdout:3999: write oid 408 current snap is 0 2014-08-08T10:52:56.314 INFO:tasks.rados.rados.0.vpm137.stdout:3999: seq_num 2436 ranges {224634=1} 2014-08-08T10:52:56.317 INFO:tasks.rados.rados.0.vpm137.stdout:3999: writing vpm1375878-408 from 224634 to 224635 tid 1 2014-08-08T10:52:56.318 INFO:tasks.rados.rados.0.vpm137.stdout:3991: finishing write tid 1 to vpm1375878-135 2014-08-08T10:52:56.318 INFO:tasks.rados.rados.0.vpm137.stdout:3991: finishing write tid 2 to vpm1375878-135 2014-08-08T10:52:56.319 INFO:tasks.rados.rados.0.vpm137.stdout:3991: finishing write tid 3 to vpm1375878-135 2014-08-08T10:52:56.319 INFO:tasks.rados.rados.0.vpm137.stdout:3991: finishing write tid 5 to vpm1375878-135 2014-08-08T10:52:56.320 INFO:tasks.rados.rados.0.vpm137.stdout:3991: finishing write tid 6 to vpm1375878-135 2014-08-08T10:52:56.320 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 135 v 365 (ObjNum 2433 snap 0 seq_num 2433) dirty exists 2014-08-08T10:52:56.320 INFO:tasks.rados.rados.0.vpm137.stdout:3992: finishing write tid 1 to vpm1375878-269 2014-08-08T10:52:56.320 INFO:tasks.rados.rados.0.vpm137.stdout:3992: finishing write tid 2 to vpm1375878-269 2014-08-08T10:52:56.321 INFO:tasks.rados.rados.0.vpm137.stdout:3992: finishing write tid 4 to vpm1375878-269 2014-08-08T10:52:56.321 INFO:tasks.rados.rados.0.vpm137.stdout:3992: finishing write tid 5 to vpm1375878-269 2014-08-08T10:52:56.322 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 269 v 423 (ObjNum 2434 snap 0 seq_num 2434) dirty exists 2014-08-08T10:52:56.322 INFO:tasks.rados.rados.0.vpm137.stdout:3993: expect (ObjNum 2395 snap 0 seq_num 2395) 2014-08-08T10:52:56.348 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:52:56.399 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:56.634 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 258, u'data_bytes': 841526296, u'bytes_used': 1688739840, u'op_per_sec': 13, u'version': 1707, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 245, u'state_name': u'active+clean'}], u'read_bytes_sec': 2488985, u'write_bytes_sec': 631533, u'bytes_avail': 427598274560}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 618, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:52:56.635 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:52:56.843 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:52:56.843 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:52:58.750 INFO:tasks.rados.rados.0.vpm137.stdout:3991: done (8 left) 2014-08-08T10:52:58.751 INFO:tasks.rados.rados.0.vpm137.stdout:3992: done (7 left) 2014-08-08T10:52:58.751 INFO:tasks.rados.rados.0.vpm137.stdout:3993: done (6 left) 2014-08-08T10:52:58.752 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 6 2014-08-08T10:52:58.752 INFO:tasks.rados.rados.0.vpm137.stdout:3995: expect (ObjNum 2021 snap 0 seq_num 2021) 2014-08-08T10:52:59.042 INFO:tasks.rados.rados.0.vpm137.stdout:3995: done (5 left) 2014-08-08T10:52:59.042 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 5 2014-08-08T10:52:59.043 INFO:tasks.rados.rados.0.vpm137.stdout:3994: expect (ObjNum 1389 snap 0 seq_num 1389) 2014-08-08T10:53:00.067 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:00.725 INFO:tasks.rados.rados.0.vpm137.stdout:3994: done (4 left) 2014-08-08T10:53:00.726 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 4 2014-08-08T10:53:00.726 INFO:tasks.rados.rados.0.vpm137.stdout:3996: finishing write tid 1 to vpm1375878-160 2014-08-08T10:53:00.726 INFO:tasks.rados.rados.0.vpm137.stdout:3996: finishing write tid 2 to vpm1375878-160 2014-08-08T10:53:00.728 INFO:tasks.rados.rados.0.vpm137.stdout:3996: finishing write tid 3 to vpm1375878-160 2014-08-08T10:53:00.728 INFO:tasks.rados.rados.0.vpm137.stdout:3996: finishing write tid 5 to vpm1375878-160 2014-08-08T10:53:00.728 INFO:tasks.rados.rados.0.vpm137.stdout:3996: finishing write tid 6 to vpm1375878-160 2014-08-08T10:53:00.728 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 160 v 427 (ObjNum 2435 snap 0 seq_num 2435) dirty exists 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout:3996: done (3 left) 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 3 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout:3998: done (2 left) 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 2 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout:3999: finishing write tid 1 to vpm1375878-408 2014-08-08T10:53:00.729 INFO:tasks.rados.rados.0.vpm137.stdout:3999: finishing write tid 3 to vpm1375878-408 2014-08-08T10:53:00.730 INFO:tasks.rados.rados.0.vpm137.stdout:3999: finishing write tid 4 to vpm1375878-408 2014-08-08T10:53:00.730 INFO:tasks.rados.rados.0.vpm137.stdout:update_object_version oid 408 v 384 (ObjNum 2436 snap 0 seq_num 2436) dirty exists 2014-08-08T10:53:00.730 INFO:tasks.rados.rados.0.vpm137.stdout:3999: done (1 left) 2014-08-08T10:53:00.730 INFO:tasks.rados.rados.0.vpm137.stdout: waiting on 1 2014-08-08T10:53:00.730 INFO:tasks.rados.rados.0.vpm137.stdout:3997: expect (ObjNum 1724 snap 0 seq_num 1724) 2014-08-08T10:53:02.851 INFO:tasks.rados.rados.0.vpm137.stdout:3997: done (0 left) 2014-08-08T10:53:03.120 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:03.317 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:03.373 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:03.615 INFO:tasks.rados.rados.0.vpm137.stderr:0 errors. 2014-08-08T10:53:03.616 INFO:tasks.rados.rados.0.vpm137.stderr: 2014-08-08T10:53:04.002 INFO:tasks.thrashosds.ceph_manager:removing pool_name unique_pool_0 2014-08-08T10:53:04.003 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados rmpool unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it' 2014-08-08T10:53:04.173 INFO:teuthology.orchestra.run.vpm135.stdout:successfully deleted pool unique_pool_0 2014-08-08T10:53:06.618 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:52:13.454645', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89723528, u'health': u'HEALTH_OK', u'kb_used': 9230040, u'store_stats': {u'bytes_total': 22261054, u'bytes_log': 413978, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 1202126848, u'version': 1710, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428084887552}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:06.619 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:09.832 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:09.833 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:13.045 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:16.073 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:16.280 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:16.320 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:16.553 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:53:13.455139', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712788, u'health': u'HEALTH_OK', u'kb_used': 9240780, u'store_stats': {u'bytes_total': 23013528, u'bytes_log': 1166452, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:52:17.562907', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89722404, u'health': u'HEALTH_OK', u'kb_used': 9231164, u'store_stats': {u'bytes_total': 22526339, u'bytes_log': 679319, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 507650048, u'version': 1711, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428779364352}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:16.554 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:16.753 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:16.753 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:16.945 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:19.974 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:20.172 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:20.217 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:23.445 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:53:13.455139', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712788, u'health': u'HEALTH_OK', u'kb_used': 9240780, u'store_stats': {u'bytes_total': 23013528, u'bytes_log': 1166452, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:53:17.563367', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712712, u'health': u'HEALTH_OK', u'kb_used': 9240856, u'store_stats': {u'bytes_total': 23590168, u'bytes_log': 1743148, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 507650048, u'version': 1712, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428779364352}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:23.446 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:23.650 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:23.650 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:26.872 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:29.904 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:30.113 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:30.171 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:30.407 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:53:13.455139', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712788, u'health': u'HEALTH_OK', u'kb_used': 9240780, u'store_stats': {u'bytes_total': 23013528, u'bytes_log': 1166452, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:53:17.563367', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712712, u'health': u'HEALTH_OK', u'kb_used': 9240856, u'store_stats': {u'bytes_total': 23590168, u'bytes_log': 1743148, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 507650048, u'version': 1712, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428779364352}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:30.408 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:33.629 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:33.630 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:36.841 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:39.863 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:40.069 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:40.115 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:40.343 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:53:13.455139', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712788, u'health': u'HEALTH_OK', u'kb_used': 9240780, u'store_stats': {u'bytes_total': 23013528, u'bytes_log': 1166452, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:53:17.563367', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712712, u'health': u'HEALTH_OK', u'kb_used': 9240856, u'store_stats': {u'bytes_total': 23590168, u'bytes_log': 1743148, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 507650048, u'version': 1712, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428779364352}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:40.344 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:40.539 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:40.539 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:40.737 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:43.766 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:43.980 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:44.024 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:44.277 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 56, u'quorum': [1, 2], u'mdsmap': {u'max': 1, u'epoch': 6, u'by_rank': [{u'status': u'up:active(laggy or crashed)', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.155:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.140.187:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.140.187:6790/0'}], u'modified': u'2014-08-08 16:54:21.349412', u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf', u'created': u'2014-08-08 16:54:21.349412'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 56, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013433', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 17:53:13.455139', u'name': u'a', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712788, u'health': u'HEALTH_OK', u'kb_used': 9240780, u'store_stats': {u'bytes_total': 23013528, u'bytes_log': 1166452, u'last_updated': u'0.000000', u'bytes_misc': 21847076, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 17:53:17.563367', u'name': u'b', u'avail_percent': 86, u'kb_total': 103178528, u'kb_avail': 89712712, u'health': u'HEALTH_OK', u'kb_used': 9240856, u'store_stats': {u'bytes_total': 23590168, u'bytes_log': 1743148, u'last_updated': u'0.000000', u'bytes_misc': 21847020, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mds a is laggy'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 mons down, quorum 1,2 a,b'}]}, u'pgmap': {u'bytes_total': 429287014400, u'num_pgs': 242, u'data_bytes': 80274, u'bytes_used': 507650048, u'version': 1712, u'pgs_by_state': [{u'count': 13, u'state_name': u'creating'}, {u'count': 229, u'state_name': u'active+clean'}], u'bytes_avail': 428779364352}, u'quorum_names': [u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 3, u'epoch': 619, u'num_in_osds': 2}}, u'fsid': u'0164d489-27e1-40f7-a1c0-e7a28750b5bf'} 2014-08-08T10:53:44.278 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T10:53:44.503 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T10:53:44.503 INFO:teuthology.orchestra.run.vpm135:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T10:53:44.752 INFO:teuthology.orchestra.run.vpm135.stderr:dumped all in format json 2014-08-08T10:53:46.280 INFO:teuthology.orchestra.run.vpm137.stderr:testContainerSerializedInfo (test.functional.tests.TestAccount) ... ERROR ---------------------------------------------------------------------- 2014-08-08T11:40:24.096 INFO:teuthology.orchestra.run.vpm146.stderr:testContainerNameLimit (test.functional.tests.TestContainerUTF8) ... ok 2014-08-08T11:40:26.999 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:40:27.441 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:40:27.564 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:40:27.847 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.109924', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.122711', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:40:22.077602', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771132, u'health': u'HEALTH_OK', u'kb_used': 16182436, u'store_stats': {u'bytes_total': 50558071, u'bytes_log': 15267723, u'last_updated': u'0.000000', u'bytes_misc': 35290348, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:39:31.889776', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83391596, u'health': u'HEALTH_OK', u'kb_used': 15561972, u'store_stats': {u'bytes_total': 51031748, u'bytes_log': 15630799, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:39:38.758540', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83390304, u'health': u'HEALTH_OK', u'kb_used': 15563264, u'store_stats': {u'bytes_total': 51289775, u'bytes_log': 15888826, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 23/428 objects degraded (5.374%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 50488, u'degraded_objects': 23, u'num_pgs': 806, u'recovering_keys_per_sec': 6, u'data_bytes': 506955, u'degraded_total': 428, u'bytes_used': 687460352, u'recovering_objects_per_sec': 14, u'write_bytes_sec': 53132, u'op_per_sec': 32, u'version': 4440, u'pgs_by_state': [{u'count': 1, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'active'}, {u'count': 2, u'state_name': u'peering'}, {u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 798, u'state_name': u'active+clean'}], u'read_bytes_sec': 6214, u'degraded_ratio': u'5.374', u'bytes_avail': 857886568448}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2508, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:40:27.847 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:40:28.108 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:40:28.108 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:40:28.432 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:40:31.512 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:40:31.799 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:40:31.907 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:40:32.393 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.109924', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.122711', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:40:22.077602', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771132, u'health': u'HEALTH_OK', u'kb_used': 16182436, u'store_stats': {u'bytes_total': 50558071, u'bytes_log': 15267723, u'last_updated': u'0.000000', u'bytes_misc': 35290348, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:39:38.758540', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83390304, u'health': u'HEALTH_OK', u'kb_used': 15563264, u'store_stats': {u'bytes_total': 51289775, u'bytes_log': 15888826, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 1066, u'num_pgs': 806, u'recovering_keys_per_sec': 0, u'data_bytes': 508275, u'bytes_used': 687796224, u'recovering_objects_per_sec': 4, u'op_per_sec': 9, u'version': 4442, u'pgs_by_state': [{u'count': 806, u'state_name': u'active+clean'}], u'read_bytes_sec': 5051, u'write_bytes_sec': 1010, u'bytes_avail': 857886232576}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2508, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:40:32.394 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T11:40:32.394 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:40:32.394 INFO:tasks.thrashosds.thrasher:Adding osd 0 2014-08-08T11:40:32.394 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 0' 2014-08-08T11:40:33.233 INFO:teuthology.orchestra.run.vpm144.stderr:marked in osd.0. 2014-08-08T11:40:33.257 INFO:tasks.thrashosds.thrasher:Added osd 0 2014-08-08T11:40:38.257 INFO:tasks.thrashosds.thrasher:in_osds: [5, 2, 1, 3, 0] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:40:38.257 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:40:38.257 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [5, 2, 1, 3, 0] 2014-08-08T11:40:38.258 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-08-08T11:40:40.254 INFO:teuthology.orchestra.run.vpm144.stderr:marked out osd.2. 2014-08-08T11:40:45.318 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:40:45.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:40:45.319 INFO:tasks.thrashosds.thrasher:fixing pg num pool metadata 2014-08-08T11:40:45.319 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:40:48.262 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:40:48.509 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 154' 2014-08-08T11:40:50.465 INFO:teuthology.orchestra.run.vpm144.stderr:set pool 1 pgp_num to 154 2014-08-08T11:40:55.514 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:40:55.515 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T11:40:56.933 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T11:40:56.981 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T11:40:58.319 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T11:40:58.359 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T11:41:00.634 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T11:41:00.694 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-08T11:41:02.464 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.0 to 1 (8655362) 2014-08-08T11:41:02.517 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T11:41:02.517 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:03.458 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:03.649 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:04.649 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:04.879 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:05.559 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.109924', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.122711', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:40:22.077602', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771132, u'health': u'HEALTH_OK', u'kb_used': 16182436, u'store_stats': {u'bytes_total': 50558071, u'bytes_log': 15267723, u'last_updated': u'0.000000', u'bytes_misc': 35290348, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'229 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'190 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'190 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 60/372 objects degraded (16.129%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 146602, u'degraded_objects': 60, u'num_pgs': 806, u'recovering_keys_per_sec': 148, u'data_bytes': 505511, u'degraded_total': 372, u'bytes_used': 698855424, u'recovering_objects_per_sec': 36, u'write_bytes_sec': 66237, u'op_per_sec': 1423, u'version': 4461, u'pgs_by_state': [{u'count': 185, u'state_name': u'inactive'}, {u'count': 4, u'state_name': u'active'}, {u'count': 229, u'state_name': u'peering'}, {u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 380, u'state_name': u'active+clean'}], u'read_bytes_sec': 893962, u'degraded_ratio': u'16.129', u'bytes_avail': 857875173376}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:05.559 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:06.486 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:41:06.486 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:07.010 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:10.246 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:10.879 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:11.127 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:11.366 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.109924', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.122711', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:40:22.077602', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771132, u'health': u'HEALTH_OK', u'kb_used': 16182436, u'store_stats': {u'bytes_total': 50558071, u'bytes_log': 15267723, u'last_updated': u'0.000000', u'bytes_misc': 35290348, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'225 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'177 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'188 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 32/372 objects degraded (8.602%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 13945, u'degraded_objects': 32, u'num_pgs': 806, u'recovering_keys_per_sec': 58, u'data_bytes': 505511, u'degraded_total': 372, u'bytes_used': 689651712, u'recovering_objects_per_sec': 6, u'version': 4464, u'pgs_by_state': [{u'count': 144, u'state_name': u'inactive'}, {u'count': 11, u'state_name': u'active'}, {u'count': 225, u'state_name': u'peering'}, {u'count': 7, u'state_name': u'active+recovery_wait'}, {u'count': 418, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'degraded_ratio': u'8.602', u'bytes_avail': 857884377088}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:11.367 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:13.630 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:41:13.630 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:15.217 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:18.693 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:19.434 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:19.575 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:20.102 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:40:22.077602', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771132, u'health': u'HEALTH_OK', u'kb_used': 16182436, u'store_stats': {u'bytes_total': 50558071, u'bytes_log': 15267723, u'last_updated': u'0.000000', u'bytes_misc': 35290348, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'233 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'160 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'166 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 32/372 objects degraded (8.602%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'degraded_objects': 32, u'num_pgs': 806, u'data_bytes': 505511, u'degraded_total': 372, u'bytes_used': 682975232, u'version': 4469, u'pgs_by_state': [{u'count': 58, u'state_name': u'inactive'}, {u'count': 6, u'state_name': u'active'}, {u'count': 233, u'state_name': u'peering'}, {u'count': 7, u'state_name': u'active+recovery_wait'}, {u'count': 501, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'degraded_ratio': u'8.602', u'bytes_avail': 857891053568}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:20.103 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:20.661 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:41:20.661 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:21.167 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:24.247 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:24.764 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:24.943 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:25.402 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:41:22.078216', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82713380, u'health': u'HEALTH_OK', u'kb_used': 16240188, u'store_stats': {u'bytes_total': 46373849, u'bytes_log': 3115412, u'last_updated': u'0.000000', u'bytes_misc': 43258437, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'234 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'155 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'155 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21/358 objects degraded (5.866%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 17606, u'degraded_objects': 21, u'num_pgs': 806, u'recovering_keys_per_sec': 0, u'data_bytes': 479804, u'degraded_total': 358, u'bytes_used': 679010304, u'recovering_objects_per_sec': 3, u'version': 4473, u'pgs_by_state': [{u'count': 6, u'state_name': u'inactive'}, {u'count': 7, u'state_name': u'active'}, {u'count': 234, u'state_name': u'peering'}, {u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 557, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'degraded_ratio': u'5.866', u'bytes_avail': 857895018496}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:25.402 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:25.902 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:41:25.902 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:26.328 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:29.396 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:29.979 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:30.131 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:31.471 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:41:22.078216', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82713380, u'health': u'HEALTH_OK', u'kb_used': 16240188, u'store_stats': {u'bytes_total': 46373849, u'bytes_log': 3115412, u'last_updated': u'0.000000', u'bytes_misc': 43258437, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:31.890376', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83296348, u'health': u'HEALTH_OK', u'kb_used': 15657220, u'store_stats': {u'bytes_total': 60091082, u'bytes_log': 24690133, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'212 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'143 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'143 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21/358 objects degraded (5.866%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'degraded_objects': 21, u'num_pgs': 806, u'data_bytes': 479804, u'degraded_total': 358, u'bytes_used': 682647552, u'version': 4476, u'pgs_by_state': [{u'count': 7, u'state_name': u'active'}, {u'count': 212, u'state_name': u'peering'}, {u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 584, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'degraded_ratio': u'5.866', u'bytes_avail': 857891381248}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:31.472 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:31.997 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T11:41:31.997 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:32.774 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:35.919 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:36.333 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:36.447 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:37.636 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:41:22.078216', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82713380, u'health': u'HEALTH_OK', u'kb_used': 16240188, u'store_stats': {u'bytes_total': 46373849, u'bytes_log': 3115412, u'last_updated': u'0.000000', u'bytes_misc': 43258437, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:41:31.890884', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83202008, u'health': u'HEALTH_OK', u'kb_used': 15751560, u'store_stats': {u'bytes_total': 48350019, u'bytes_log': 4980914, u'last_updated': u'0.000000', u'bytes_misc': 43369105, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:40:38.759031', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83281620, u'health': u'HEALTH_OK', u'kb_used': 15671948, u'store_stats': {u'bytes_total': 60371895, u'bytes_log': 24970946, u'last_updated': u'0.000000', u'bytes_misc': 35400949, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'recovery 26/436 objects degraded (5.963%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 77931, u'degraded_objects': 26, u'num_pgs': 806, u'recovering_keys_per_sec': 65, u'data_bytes': 508275, u'degraded_total': 436, u'bytes_used': 677785600, u'recovering_objects_per_sec': 31, u'write_bytes_sec': 217200, u'op_per_sec': 200, u'version': 4480, u'pgs_by_state': [{u'count': 2, u'state_name': u'active'}, {u'count': 804, u'state_name': u'active+clean'}], u'read_bytes_sec': 39624, u'degraded_ratio': u'5.963', u'bytes_avail': 857896243200}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:37.637 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:38.014 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:41:38.014 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:38.531 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:41.599 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:42.173 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:42.298 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:41:42.869 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:41:22.078216', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82713380, u'health': u'HEALTH_OK', u'kb_used': 16240188, u'store_stats': {u'bytes_total': 46373849, u'bytes_log': 3115412, u'last_updated': u'0.000000', u'bytes_misc': 43258437, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:41:31.890884', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83202008, u'health': u'HEALTH_OK', u'kb_used': 15751560, u'store_stats': {u'bytes_total': 48350019, u'bytes_log': 4980914, u'last_updated': u'0.000000', u'bytes_misc': 43369105, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:41:38.759490', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83186748, u'health': u'HEALTH_OK', u'kb_used': 15766820, u'store_stats': {u'bytes_total': 48807965, u'bytes_log': 5438860, u'last_updated': u'0.000000', u'bytes_misc': 43369105, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}]}, u'pgmap': {u'bytes_total': 858574028800, u'recovering_bytes_per_sec': 70502, u'num_pgs': 806, u'recovering_keys_per_sec': 51, u'data_bytes': 508275, u'bytes_used': 678858752, u'recovering_objects_per_sec': 27, u'op_per_sec': 156, u'version': 4481, u'pgs_by_state': [{u'count': 806, u'state_name': u'active+clean'}], u'read_bytes_sec': 31001, u'write_bytes_sec': 169932, u'bytes_avail': 857895170048}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2517, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:41:42.870 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T11:41:42.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:41:42.870 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2014-08-08T11:41:42.871 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 1 0' 2014-08-08T11:41:43.427 INFO:teuthology.orchestra.run.vpm144.stderr:set osd.1 primary-affinity to 0 (802) 2014-08-08T11:41:48.466 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:41:48.466 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:41:48.466 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-08T11:41:48.467 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:41:49.998 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:41:50.239 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T11:41:50.240 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 224' 2014-08-08T11:41:51.933 INFO:teuthology.orchestra.run.vpm144.stderr:set pool 2 pg_num to 224 2014-08-08T11:41:56.972 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:41:56.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:41:56.973 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.8771310937 2014-08-08T11:41:56.973 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 0.8771310937' 2014-08-08T11:41:57.477 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.0 to 0.877131 (8574832) 2014-08-08T11:42:02.528 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:02.528 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:02.529 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-08T11:42:02.529 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:42:04.801 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:42:09.893 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:09.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:09.894 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-08T11:42:09.894 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T11:42:09.894 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T11:42:09.894 INFO:teuthology.orchestra.run.vpm145:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-08T11:42:14.984 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:14.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:14.985 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-08T11:42:14.985 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T11:42:14.985 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T11:42:14.986 INFO:teuthology.orchestra.run.vpm145:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-08T11:42:20.079 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:20.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:20.080 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-08T11:42:20.080 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:42:21.773 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:42:26.945 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0] out_osds: [4, 2] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:26.945 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T11:42:28.154 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T11:42:28.196 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T11:42:29.193 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T11:42:29.238 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T11:42:31.142 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T11:42:31.204 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-08T11:42:32.462 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.0 to 1 (8655362) 2014-08-08T11:42:32.513 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T11:42:32.514 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:42:33.344 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:42:33.533 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:42:34.318 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:42:34.446 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:42:34.979 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:42:22.078723', u'name': u'c', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82619692, u'health': u'HEALTH_OK', u'kb_used': 16333876, u'store_stats': {u'bytes_total': 43968395, u'bytes_log': 820070, u'last_updated': u'0.000000', u'bytes_misc': 43148325, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:42:31.891399', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83078196, u'health': u'HEALTH_OK', u'kb_used': 15875372, u'store_stats': {u'bytes_total': 45306054, u'bytes_log': 2047032, u'last_updated': u'0.000000', u'bytes_misc': 43259022, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:41:38.759490', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 83186748, u'health': u'HEALTH_OK', u'kb_used': 15766820, u'store_stats': {u'bytes_total': 48807965, u'bytes_log': 5438860, u'last_updated': u'0.000000', u'bytes_misc': 43369105, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 294 > pgp_num 234'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 858574028800, u'num_pgs': 816, u'data_bytes': 508275, u'bytes_used': 679600128, u'version': 4511, u'pgs_by_state': [{u'count': 815, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}], u'bytes_avail': 857894428672}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2532, u'num_in_osds': 4}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:42:34.980 INFO:tasks.thrashosds.ceph_manager:recovered! 2014-08-08T11:42:34.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:34.980 INFO:tasks.thrashosds.thrasher:Adding osd 2 2014-08-08T11:42:34.981 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-08-08T11:42:38.234 INFO:teuthology.orchestra.run.vpm144.stderr:marked in osd.2. 2014-08-08T11:42:38.291 INFO:tasks.thrashosds.thrasher:Added osd 2 2014-08-08T11:42:43.290 INFO:tasks.thrashosds.thrasher:in_osds: [5, 1, 3, 0, 2] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:43.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:43.291 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [5, 1, 3, 0, 2] 2014-08-08T11:42:43.291 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-08-08T11:42:45.330 INFO:teuthology.orchestra.run.vpm144.stderr:marked out osd.5. 2014-08-08T11:42:50.382 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2] out_osds: [4, 5] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:50.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:50.382 INFO:tasks.thrashosds.thrasher:fixing pg num pool data 2014-08-08T11:42:50.382 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:42:51.396 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:42:51.578 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 294' 2014-08-08T11:42:54.778 INFO:teuthology.orchestra.run.vpm144.stderr:set pool 0 pgp_num to 294 2014-08-08T11:42:59.802 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2] out_osds: [4, 5] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:42:59.802 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:42:59.802 INFO:tasks.thrashosds.thrasher:Adding osd 5 2014-08-08T11:42:59.803 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-08-08T11:43:01.896 INFO:teuthology.orchestra.run.vpm144.stderr:marked in osd.5. 2014-08-08T11:43:01.958 INFO:tasks.thrashosds.thrasher:Added osd 5 2014-08-08T11:43:06.958 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:06.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:06.958 INFO:tasks.thrashosds.thrasher:Growing pool data 2014-08-08T11:43:06.959 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:43:09.787 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:43:09.975 INFO:tasks.thrashosds.ceph_manager:increase pool size by 10 2014-08-08T11:43:09.976 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 304' 2014-08-08T11:43:14.305 INFO:teuthology.orchestra.run.vpm144.stderr:set pool 0 pg_num to 304 2014-08-08T11:43:19.356 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:19.356 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:19.356 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.203660702773 2014-08-08T11:43:19.357 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 0.203660702773' 2014-08-08T11:43:20.190 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.5 to 0.203661 (8133472) 2014-08-08T11:43:25.250 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:25.251 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:25.251 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.974836 2014-08-08T11:43:25.251 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 2 0.974835883354' 2014-08-08T11:43:27.137 INFO:teuthology.orchestra.run.vpm144.stderr:set osd.2 primary-affinity to 0.974836 (8638862) 2014-08-08T11:43:32.169 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:32.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:32.170 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-08T11:43:32.170 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T11:43:32.170 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T11:43:32.171 INFO:teuthology.orchestra.run.vpm145:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-08T11:43:38.486 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:38.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:38.486 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-08T11:43:38.486 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T11:43:38.486 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T11:43:38.487 INFO:teuthology.orchestra.run.vpm144:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-08T11:43:45.679 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:45.679 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:45.679 INFO:tasks.thrashosds.thrasher:Growing pool metadata 2014-08-08T11:43:45.680 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:43:50.298 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:43:55.436 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:43:55.436 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:43:55.437 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-08T11:43:55.437 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-08T11:43:55.437 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-08T11:43:55.437 INFO:teuthology.orchestra.run.vpm144:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-08T11:44:00.975 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 0, 2, 5] out_osds: [4] dead_osds: [] live_osds: [5, 0, 2, 1, 3, 4] 2014-08-08T11:44:00.975 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-08T11:44:02.909 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.1 to 1 (8655362) 2014-08-08T11:44:02.950 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-08T11:44:04.046 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.3 to 1 (8655362) 2014-08-08T11:44:04.123 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-08T11:44:05.293 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.0 to 1 (8655362) 2014-08-08T11:44:05.333 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-08T11:44:10.486 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.2 to 1 (8655362) 2014-08-08T11:44:10.558 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-08T11:44:13.201 INFO:teuthology.orchestra.run.vpm144.stderr:reweighted osd.5 to 1 (8655362) 2014-08-08T11:44:13.247 INFO:tasks.thrashosds.ceph_manager:waiting for recovery to complete 2014-08-08T11:44:13.247 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:14.304 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:14.536 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:17.483 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:17.716 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:18.549 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:43:22.079131', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82482056, u'health': u'HEALTH_OK', u'kb_used': 16471512, u'store_stats': {u'bytes_total': 48469802, u'bytes_log': 5321477, u'last_updated': u'0.000000', u'bytes_misc': 43148325, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:43:31.891851', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82935120, u'health': u'HEALTH_OK', u'kb_used': 16018448, u'store_stats': {u'bytes_total': 43148497, u'bytes_log': 1014315, u'last_updated': u'0.000000', u'bytes_misc': 42134182, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:43:38.832070', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82916868, u'health': u'HEALTH_OK', u'kb_used': 16036700, u'store_stats': {u'bytes_total': 44404659, u'bytes_log': 2270477, u'last_updated': u'0.000000', u'bytes_misc': 42134182, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs backfill'}, {u'severity': u'HEALTH_WARN', u'summary': u'278 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'159 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'171 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 49/322 objects degraded (15.217%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 21907, u'degraded_objects': 49, u'num_pgs': 826, u'recovering_keys_per_sec': 0, u'data_bytes': 212678, u'degraded_total': 322, u'bytes_used': 853786624, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 126634, u'op_per_sec': 80, u'version': 4583, u'pgs_by_state': [{u'count': 121, u'state_name': u'inactive'}, {u'count': 271, u'state_name': u'peering'}, {u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 402, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+remapped+wait_backfill'}, {u'count': 1, u'state_name': u'active+recovering'}, {u'count': 7, u'state_name': u'remapped+peering'}], u'read_bytes_sec': 14355, u'degraded_ratio': u'15.217', u'bytes_avail': 1072363749376}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2575, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:18.550 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:20.264 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:44:20.264 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:20.879 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:23.934 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:25.122 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:25.334 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:26.457 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:43:31.891851', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82935120, u'health': u'HEALTH_OK', u'kb_used': 16018448, u'store_stats': {u'bytes_total': 43148497, u'bytes_log': 1014315, u'last_updated': u'0.000000', u'bytes_misc': 42134182, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:43:38.832070', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82916868, u'health': u'HEALTH_OK', u'kb_used': 16036700, u'store_stats': {u'bytes_total': 44404659, u'bytes_log': 2270477, u'last_updated': u'0.000000', u'bytes_misc': 42134182, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs backfill'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs backfilling'}, {u'severity': u'HEALTH_WARN', u'summary': u'305 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'140 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'152 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 38/326 objects degraded (11.656%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'degraded_objects': 38, u'num_pgs': 826, u'data_bytes': 212678, u'degraded_total': 326, u'bytes_used': 858124288, u'version': 4590, u'pgs_by_state': [{u'count': 107, u'state_name': u'inactive'}, {u'count': 299, u'state_name': u'peering'}, {u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 391, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+remapped+wait_backfill'}, {u'count': 6, u'state_name': u'remapped+peering'}, {u'count': 5, u'state_name': u'active+remapped+backfilling'}], u'degraded_ratio': u'11.656', u'bytes_avail': 1072359411712}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2577, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:26.457 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:28.034 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T11:44:28.034 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:29.298 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:32.491 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:34.638 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:34.978 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:36.075 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:43:38.832070', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82916868, u'health': u'HEALTH_OK', u'kb_used': 16036700, u'store_stats': {u'bytes_total': 44404659, u'bytes_log': 2270477, u'last_updated': u'0.000000', u'bytes_misc': 42134182, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs backfilling'}, {u'severity': u'HEALTH_WARN', u'summary': u'269 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'129 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'141 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 14/326 objects degraded (4.294%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 0, u'degraded_objects': 14, u'num_pgs': 826, u'recovering_keys_per_sec': 68, u'data_bytes': 212678, u'degraded_total': 326, u'bytes_used': 861200384, u'recovering_objects_per_sec': 1, u'version': 4596, u'pgs_by_state': [{u'count': 106, u'state_name': u'inactive'}, {u'count': 263, u'state_name': u'peering'}, {u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 441, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+remapped'}, {u'count': 6, u'state_name': u'remapped+peering'}, {u'count': 1, u'state_name': u'active+remapped+backfilling'}], u'degraded_ratio': u'4.294', u'bytes_avail': 1072356335616}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2580, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:36.076 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:36.940 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:44:36.941 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:37.594 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:40.698 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:43.588 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:43.806 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:44.520 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs backfilling'}, {u'severity': u'HEALTH_WARN', u'summary': u'227 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'130 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'142 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 16/324 objects degraded (4.938%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 2501, u'degraded_objects': 16, u'num_pgs': 826, u'recovering_keys_per_sec': 0, u'data_bytes': 212286, u'degraded_total': 324, u'bytes_used': 860315648, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 172172, u'op_per_sec': 426, u'version': 4600, u'pgs_by_state': [{u'count': 102, u'state_name': u'inactive'}, {u'count': 225, u'state_name': u'peering'}, {u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 485, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+remapped'}, {u'count': 2, u'state_name': u'remapped+peering'}, {u'count': 1, u'state_name': u'active+remapped+backfilling'}], u'read_bytes_sec': 635830, u'degraded_ratio': u'4.938', u'bytes_avail': 1072357220352}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2582, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:44.521 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:45.539 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:44:45.539 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:46.407 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:49.566 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:50.454 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:50.696 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:51.362 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'182 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'108 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'112 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 16/336 objects degraded (4.762%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'degraded_objects': 16, u'num_pgs': 826, u'data_bytes': 237993, u'degraded_total': 336, u'bytes_used': 864661504, u'version': 4604, u'pgs_by_state': [{u'count': 72, u'state_name': u'inactive'}, {u'count': 180, u'state_name': u'peering'}, {u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 565, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+remapped'}, {u'count': 2, u'state_name': u'remapped+peering'}], u'degraded_ratio': u'4.762', u'bytes_avail': 1072352874496}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2583, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:51.363 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:52.086 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T11:44:52.087 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:52.624 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:55.688 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:57.866 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:44:58.036 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:58.607 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'182 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'108 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'112 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 16/336 objects degraded (4.762%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'degraded_objects': 16, u'num_pgs': 826, u'data_bytes': 237993, u'degraded_total': 336, u'bytes_used': 864776192, u'version': 4606, u'pgs_by_state': [{u'count': 72, u'state_name': u'inactive'}, {u'count': 180, u'state_name': u'peering'}, {u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 565, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+remapped'}, {u'count': 2, u'state_name': u'remapped+peering'}], u'degraded_ratio': u'4.762', u'bytes_avail': 1072352759808}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2583, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:44:58.609 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:44:59.175 INFO:tasks.thrashosds.ceph_manager:no progress seen, keeping timeout for now 2014-08-08T11:44:59.176 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:44:59.890 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:03.006 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:03.756 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:03.937 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:04.384 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'167 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'101 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'105 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 16/370 objects degraded (4.324%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 19782, u'degraded_objects': 16, u'num_pgs': 826, u'recovering_keys_per_sec': 0, u'data_bytes': 335778, u'degraded_total': 370, u'bytes_used': 859279360, u'recovering_objects_per_sec': 7, u'write_bytes_sec': 53326, u'op_per_sec': 72, u'version': 4608, u'pgs_by_state': [{u'count': 88, u'state_name': u'inactive'}, {u'count': 165, u'state_name': u'peering'}, {u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 565, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'remapped+peering'}], u'read_bytes_sec': 9012, u'degraded_ratio': u'4.324', u'bytes_avail': 1072358256640}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2583, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:45:04.385 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:05.205 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:45:05.206 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:07.235 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:10.300 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:11.273 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:11.488 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:12.412 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'157 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'98 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'99 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 32/374 objects degraded (8.556%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 1306, u'degraded_objects': 32, u'num_pgs': 826, u'recovering_keys_per_sec': 0, u'data_bytes': 335862, u'degraded_total': 374, u'bytes_used': 852807680, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 25116, u'op_per_sec': 29, u'version': 4614, u'pgs_by_state': [{u'count': 59, u'state_name': u'inactive'}, {u'count': 3, u'state_name': u'active'}, {u'count': 157, u'state_name': u'peering'}, {u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 602, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 1, u'state_name': u'remapped'}], u'read_bytes_sec': 5581, u'degraded_ratio': u'8.556', u'bytes_avail': 1072364728320}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2585, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:45:12.414 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:12.953 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:45:12.954 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:13.769 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:16.921 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:17.663 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:17.809 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:18.526 INFO:tasks.thrashosds.ceph_manager:{u'election_epoch': 10, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 4, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'c', u'rank': 0, u'addr': u'10.214.139.152:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.139.189:6789/0'}, {u'name': u'b', u'rank': 2, u'addr': u'10.214.139.189:6790/0'}], u'modified': u'2014-08-08 16:51:14.531900', u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2', u'created': u'2014-08-08 16:51:14.531900'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 10, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}, {u'latency': u'0.088625', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.099046', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-08 18:44:22.079723', u'name': u'c', u'avail_percent': 79, u'kb_total': 103178528, u'kb_avail': 82401508, u'health': u'HEALTH_OK', u'kb_used': 16552060, u'store_stats': {u'bytes_total': 38566887, u'bytes_log': 3779430, u'last_updated': u'0.000000', u'bytes_misc': 34787457, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:31.892244', u'name': u'a', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82790044, u'health': u'HEALTH_OK', u'kb_used': 16163524, u'store_stats': {u'bytes_total': 42064868, u'bytes_log': 7166647, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}, {u'last_updated': u'2014-08-08 18:44:38.832581', u'name': u'b', u'avail_percent': 80, u'kb_total': 103178528, u'kb_avail': 82771228, u'health': u'HEALTH_OK', u'kb_used': 16182340, u'store_stats': {u'bytes_total': 43008060, u'bytes_log': 8109839, u'last_updated': u'0.000000', u'bytes_misc': 34898221, u'bytes_sst': 0}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'102 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'86 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'86 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 30/374 objects degraded (8.021%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 304 > pgp_num 294'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 224 > pgp_num 214'}]}, u'pgmap': {u'bytes_total': 1073217536000, u'recovering_bytes_per_sec': 52, u'degraded_objects': 30, u'num_pgs': 826, u'recovering_keys_per_sec': 0, u'data_bytes': 335862, u'degraded_total': 374, u'bytes_used': 855138304, u'recovering_objects_per_sec': 0, u'version': 4617, u'pgs_by_state': [{u'count': 66, u'state_name': u'inactive'}, {u'count': 6, u'state_name': u'active'}, {u'count': 102, u'state_name': u'peering'}, {u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 648, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+clean+scrubbing'}, {u'count': 1, u'state_name': u'remapped'}], u'degraded_ratio': u'8.021', u'bytes_avail': 1072362397696}, u'quorum_names': [u'c', u'a', u'b'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 2585, u'num_in_osds': 5}}, u'fsid': u'8f46827a-e42a-41d0-9111-a16996a3c4b2'} 2014-08-08T11:45:18.527 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:19.185 INFO:tasks.thrashosds.ceph_manager:making progress, resetting timeout 2014-08-08T11:45:19.185 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:20.072 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:23.177 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-08T11:45:23.667 INFO:teuthology.orchestra.run.vpm144.stderr:dumped all in format json 2014-08-08T11:45:23.833 INFO:teuthology.orchestra.run.vpm144:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-08T11:45:24.199 INFO:teuthology.orchestra.run.vpm146.stderr:testCreate (test.functional.tests.TestContainerUTF8) ... FAIL