2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:482: finishing write tid 3 to mira09725123-482 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:482: finishing write tid 5 to mira09725123-482 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:481: finishing write tid 1 to mira09725123-481 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:481: finishing write tid 2 to mira09725123-481 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:484: done (15 left) 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:485: done (14 left) 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:486: done (13 left) 2014-08-22 19:23:54,107.107 INFO:tasks.rados.rados.0.mira097.stdout:488: write initial oid 488 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:488: seq_num 487 ranges {495414=657098,1895437=656781,2895413=1} 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:488: writing mira09725123-488 from 495414 to 1152512 tid 1 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:488: writing mira09725123-488 from 1895437 to 2552218 tid 2 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:488: writing mira09725123-488 from 2895413 to 2895414 tid 3 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:489: write initial oid 489 2014-08-22 19:23:54,108.108 INFO:tasks.rados.rados.0.mira097.stdout:489: seq_num 488 ranges {591667=628902,2009568=582099} 2014-08-22 19:23:54,125.125 INFO:tasks.rados.rados.0.mira097.stdout:489: writing mira09725123-489 from 591667 to 1220569 tid 1 2014-08-22 19:23:54,152.152 INFO:tasks.rados.rados.0.mira097.stdout:489: writing mira09725123-489 from 2009568 to 2591667 tid 2 2014-08-22 19:23:54,156.156 INFO:tasks.rados.rados.0.mira097.stdout:490: write initial oid 490 2014-08-22 19:23:54,157.157 INFO:tasks.rados.rados.0.mira097.stdout:490: seq_num 489 ranges {687921=752131,2123701=164220} 2014-08-22 19:23:54,182.182 INFO:tasks.rados.rados.0.mira097.stdout:490: writing mira09725123-490 from 687921 to 1440052 tid 1 2014-08-22 19:23:54,193.193 INFO:tasks.rados.rados.0.mira097.stdout:490: writing mira09725123-490 from 2123701 to 2287921 tid 2 2014-08-22 19:23:54,202.202 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 16 2014-08-22 19:23:54,202.202 INFO:tasks.rados.rados.0.mira097.stdout:481: finishing write tid 4 to mira09725123-481 2014-08-22 19:23:54,202.202 INFO:tasks.rados.rados.0.mira097.stdout:482: finishing write tid 6 to mira09725123-482 2014-08-22 19:23:54,202.202 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 482 v 683 (ObjNum 481 snap 0 seq_num 481) dirty exists 2014-08-22 19:23:54,202.202 INFO:tasks.rados.rados.0.mira097.stdout:479: finishing write tid 1 to mira09725123-479 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:479: finishing write tid 2 to mira09725123-479 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:482: done (15 left) 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:491: write initial oid 491 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:491: seq_num 490 ranges {80302=1} 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:491: writing mira09725123-491 from 80302 to 80303 tid 1 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 16 2014-08-22 19:23:54,203.203 INFO:tasks.rados.rados.0.mira097.stdout:479: finishing write tid 4 to mira09725123-479 2014-08-22 19:23:54,204.204 INFO:tasks.rados.rados.0.mira097.stdout:481: finishing write tid 5 to mira09725123-481 2014-08-22 19:23:54,204.204 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 481 v 686 (ObjNum 480 snap 0 seq_num 480) dirty exists 2014-08-22 19:23:54,204.204 INFO:tasks.rados.rados.0.mira097.stdout:481: done (15 left) 2014-08-22 19:23:54,204.204 INFO:tasks.rados.rados.0.mira097.stdout:492: write initial oid 492 2014-08-22 19:23:54,204.204 INFO:tasks.rados.rados.0.mira097.stdout:492: seq_num 491 ranges {575533=731835,2048093=440410,3276424=499109} 2014-08-22 19:23:54,221.221 INFO:tasks.rados.rados.0.mira097.stdout:492: writing mira09725123-492 from 575533 to 1307368 tid 1 2014-08-22 19:23:54,243.243 INFO:tasks.rados.rados.0.mira097.stdout:492: writing mira09725123-492 from 2048093 to 2488503 tid 2 2014-08-22 19:23:54,261.261 INFO:tasks.rados.rados.0.mira097.stdout:492: writing mira09725123-492 from 3276424 to 3775533 tid 3 2014-08-22 19:23:54,261.261 INFO:tasks.rados.rados.0.mira097.stdout:476: finishing write tid 1 to mira09725123-476 2014-08-22 19:23:54,262.262 INFO:tasks.rados.rados.0.mira097.stdout:476: finishing write tid 3 to mira09725123-476 2014-08-22 19:23:54,279.279 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 1 to mira09725123-475 2014-08-22 19:23:54,279.279 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 2 to mira09725123-475 2014-08-22 19:23:54,279.279 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 3 to mira09725123-475 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 4 to mira09725123-475 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 6 to mira09725123-475 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:479: finishing write tid 5 to mira09725123-479 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 479 v 692 (ObjNum 478 snap 0 seq_num 478) dirty exists 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:476: finishing write tid 4 to mira09725123-476 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 476 v 694 (ObjNum 475 snap 0 seq_num 475) dirty exists 2014-08-22 19:23:54,280.280 INFO:tasks.rados.rados.0.mira097.stdout:487: finishing write tid 1 to mira09725123-487 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:487: finishing write tid 2 to mira09725123-487 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:455: finishing write tid 1 to mira09725123-455 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:487: finishing write tid 4 to mira09725123-487 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:475: finishing write tid 7 to mira09725123-475 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 475 v 699 (ObjNum 474 snap 0 seq_num 474) dirty exists 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:487: finishing write tid 5 to mira09725123-487 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 487 v 748 (ObjNum 486 snap 0 seq_num 486) dirty exists 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:488: finishing write tid 1 to mira09725123-488 2014-08-22 19:23:54,281.281 INFO:tasks.rados.rados.0.mira097.stdout:455: finishing write tid 2 to mira09725123-455 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:455: finishing write tid 4 to mira09725123-455 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:455: finishing write tid 5 to mira09725123-455 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 455 v 618 (ObjNum 454 snap 0 seq_num 454) dirty exists 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:488: finishing write tid 2 to mira09725123-488 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:488: finishing write tid 3 to mira09725123-488 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:488: finishing write tid 5 to mira09725123-488 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:488: finishing write tid 6 to mira09725123-488 2014-08-22 19:23:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 488 v 596 (ObjNum 487 snap 0 seq_num 487) dirty exists 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:491: finishing write tid 1 to mira09725123-491 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:491: finishing write tid 3 to mira09725123-491 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:491: finishing write tid 4 to mira09725123-491 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 491 v 604 (ObjNum 490 snap 0 seq_num 490) dirty exists 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:459: finishing write tid 1 to mira09725123-459 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:459: finishing write tid 3 to mira09725123-459 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:459: finishing write tid 4 to mira09725123-459 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 459 v 623 (ObjNum 458 snap 0 seq_num 458) dirty exists 2014-08-22 19:23:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 1 to mira09725123-474 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:455: done (15 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:459: done (14 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:475: done (13 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:476: done (12 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:479: done (11 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:487: done (10 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:488: done (9 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:491: done (8 left) 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:493: write initial oid 493 2014-08-22 19:23:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:493: seq_num 492 ranges {787114=552216,1842481=757133,3254033=514179,3987113=1} 2014-08-22 19:23:54,300.300 INFO:tasks.rados.rados.0.mira097.stdout:493: writing mira09725123-493 from 787114 to 1339330 tid 1 2014-08-22 19:23:54,330.330 INFO:tasks.rados.rados.0.mira097.stdout:493: writing mira09725123-493 from 1842481 to 2599614 tid 2 2014-08-22 19:23:54,350.350 INFO:tasks.rados.rados.0.mira097.stdout:493: writing mira09725123-493 from 3254033 to 3768212 tid 3 2014-08-22 19:23:54,352.352 INFO:tasks.rados.rados.0.mira097.stdout:493: writing mira09725123-493 from 3987113 to 3987114 tid 4 2014-08-22 19:23:54,356.356 INFO:tasks.rados.rados.0.mira097.stdout:494: write initial oid 494 2014-08-22 19:23:54,356.356 INFO:tasks.rados.rados.0.mira097.stdout:494: seq_num 493 ranges {580520=675444,1780519=1} 2014-08-22 19:23:54,375.375 INFO:tasks.rados.rados.0.mira097.stdout:494: writing mira09725123-494 from 580520 to 1255964 tid 1 2014-08-22 19:23:54,382.382 INFO:tasks.rados.rados.0.mira097.stdout:494: writing mira09725123-494 from 1780519 to 1780520 tid 2 2014-08-22 19:23:54,382.382 INFO:tasks.rados.rados.0.mira097.stdout:495: write initial oid 495 2014-08-22 19:23:54,382.382 INFO:tasks.rados.rados.0.mira097.stdout:495: seq_num 494 ranges {676774=530896,1476773=1} 2014-08-22 19:23:54,399.399 INFO:tasks.rados.rados.0.mira097.stdout:495: writing mira09725123-495 from 676774 to 1207670 tid 1 2014-08-22 19:23:54,404.404 INFO:tasks.rados.rados.0.mira097.stdout:495: writing mira09725123-495 from 1476773 to 1476774 tid 2 2014-08-22 19:23:54,404.404 INFO:tasks.rados.rados.0.mira097.stdout:496: write initial oid 496 2014-08-22 19:23:54,404.404 INFO:tasks.rados.rados.0.mira097.stdout:496: seq_num 495 ranges {772004=400000} 2014-08-22 19:23:54,418.418 INFO:tasks.rados.rados.0.mira097.stdout:496: writing mira09725123-496 from 772004 to 1172004 tid 1 2014-08-22 19:23:54,421.421 INFO:tasks.rados.rados.0.mira097.stdout:497: write initial oid 497 2014-08-22 19:23:54,422.422 INFO:tasks.rados.rados.0.mira097.stdout:497: seq_num 496 ranges {681761=626953,1809665=424410,2690358=687646,3481760=1} 2014-08-22 19:23:54,441.441 INFO:tasks.rados.rados.0.mira097.stdout:497: writing mira09725123-497 from 681761 to 1308714 tid 1 2014-08-22 19:23:54,456.456 INFO:tasks.rados.rados.0.mira097.stdout:497: writing mira09725123-497 from 1809665 to 2234075 tid 2 2014-08-22 19:23:54,479.479 INFO:tasks.rados.rados.0.mira097.stdout:497: writing mira09725123-497 from 2690358 to 3378004 tid 3 2014-08-22 19:23:54,483.483 INFO:tasks.rados.rados.0.mira097.stdout:497: writing mira09725123-497 from 3481760 to 3481761 tid 4 2014-08-22 19:23:54,486.486 INFO:tasks.rados.rados.0.mira097.stdout:498: write initial oid 498 2014-08-22 19:23:54,486.486 INFO:tasks.rados.rados.0.mira097.stdout:498: seq_num 497 ranges {776991=750181,2189525=646029,3176990=1} 2014-08-22 19:23:54,507.507 INFO:tasks.rados.rados.0.mira097.stdout:498: writing mira09725123-498 from 776991 to 1527172 tid 1 2014-08-22 19:23:54,531.531 INFO:tasks.rados.rados.0.mira097.stdout:498: writing mira09725123-498 from 2189525 to 2835554 tid 2 2014-08-22 19:23:54,537.537 INFO:tasks.rados.rados.0.mira097.stdout:498: writing mira09725123-498 from 3176990 to 3176991 tid 3 2014-08-22 19:23:54,538.538 INFO:tasks.rados.rados.0.mira097.stdout:499: write initial oid 499 2014-08-22 19:23:54,538.538 INFO:tasks.rados.rados.0.mira097.stdout:499: seq_num 498 ranges {473245=606658,1788330=716224,2873244=1} 2014-08-22 19:23:54,555.555 INFO:tasks.rados.rados.0.mira097.stdout:499: writing mira09725123-499 from 473245 to 1079903 tid 1 2014-08-22 19:23:54,580.580 INFO:tasks.rados.rados.0.mira097.stdout:499: writing mira09725123-499 from 1788330 to 2504554 tid 2 2014-08-22 19:23:54,585.585 INFO:tasks.rados.rados.0.mira097.stdout:499: writing mira09725123-499 from 2873244 to 2873245 tid 3 2014-08-22 19:23:54,586.586 INFO:tasks.rados.rados.0.mira097.stdout:500: write initial oid 500 2014-08-22 19:23:54,586.586 INFO:tasks.rados.rados.0.mira097.stdout:500: seq_num 499 ranges {665625=1} 2014-08-22 19:23:54,587.587 INFO:tasks.rados.rados.0.mira097.stdout:500: writing mira09725123-500 from 665625 to 665626 tid 1 2014-08-22 19:23:54,587.587 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 16 2014-08-22 19:23:54,590.590 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 2 to mira09725123-474 2014-08-22 19:23:54,590.590 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 3 to mira09725123-474 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 4 to mira09725123-474 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 6 to mira09725123-474 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:474: finishing write tid 7 to mira09725123-474 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 474 v 633 (ObjNum 473 snap 0 seq_num 473) dirty exists 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:473: finishing write tid 1 to mira09725123-473 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:473: finishing write tid 2 to mira09725123-473 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:492: finishing write tid 1 to mira09725123-492 2014-08-22 19:23:54,591.591 INFO:tasks.rados.rados.0.mira097.stdout:473: finishing write tid 3 to mira09725123-473 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:473: finishing write tid 5 to mira09725123-473 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:473: finishing write tid 6 to mira09725123-473 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 473 v 637 (ObjNum 472 snap 0 seq_num 472) dirty exists 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:468: finishing write tid 1 to mira09725123-468 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:468: finishing write tid 3 to mira09725123-468 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:468: finishing write tid 4 to mira09725123-468 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 468 v 639 (ObjNum 467 snap 0 seq_num 467) dirty exists 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:465: finishing write tid 1 to mira09725123-465 2014-08-22 19:23:54,592.592 INFO:tasks.rados.rados.0.mira097.stdout:465: finishing write tid 2 to mira09725123-465 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:465: finishing write tid 3 to mira09725123-465 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:468: done (15 left) 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:473: done (14 left) 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:474: done (13 left) 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:501: delete oid 237 current snap is 0 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:465: finishing write tid 5 to mira09725123-465 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:465: finishing write tid 6 to mira09725123-465 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 465 v 643 (ObjNum 464 snap 0 seq_num 464) dirty exists 2014-08-22 19:23:54,593.593 INFO:tasks.rados.rados.0.mira097.stdout:462: finishing write tid 1 to mira09725123-462 2014-08-22 19:23:54,594.594 INFO:tasks.rados.rados.0.mira097.stdout:462: finishing write tid 2 to mira09725123-462 2014-08-22 19:23:54,594.594 INFO:tasks.rados.rados.0.mira097.stdout:462: finishing write tid 4 to mira09725123-462 2014-08-22 19:23:54,594.594 INFO:tasks.rados.rados.0.mira097.stdout:462: finishing write tid 5 to mira09725123-462 2014-08-22 19:23:54,594.594 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 462 v 646 (ObjNum 461 snap 0 seq_num 461) dirty exists 2014-08-22 19:23:54,594.594 INFO:tasks.rados.rados.0.mira097.stdout:492: finishing write tid 2 to mira09725123-492 2014-08-22 19:23:54,881.881 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:23:54,907.907 INFO:tasks.rados.rados.0.mira097.stdout:492: finishing write tid 3 to mira09725123-492 2014-08-22 19:23:54,908.908 INFO:tasks.rados.rados.0.mira097.stdout:492: finishing write tid 5 to mira09725123-492 2014-08-22 19:23:54,917.917 INFO:tasks.rados.rados.0.mira097.stdout:492: finishing write tid 6 to mira09725123-492 2014-08-22 19:23:54,917.917 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 492 v 611 (ObjNum 491 snap 0 seq_num 491) dirty exists 2014-08-22 19:23:54,935.935 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 1 to mira09725123-493 2014-08-22 19:23:54,935.935 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 2 to mira09725123-493 2014-08-22 19:23:54,935.935 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 3 to mira09725123-493 2014-08-22 19:23:54,936.936 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 4 to mira09725123-493 2014-08-22 19:23:54,936.936 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 6 to mira09725123-493 2014-08-22 19:23:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:495: finishing write tid 1 to mira09725123-495 2014-08-22 19:23:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:495: finishing write tid 2 to mira09725123-495 2014-08-22 19:23:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:495: finishing write tid 4 to mira09725123-495 2014-08-22 19:23:54,938.938 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 1 to mira09725123-497 2014-08-22 19:23:54,940.940 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 2 to mira09725123-497 2014-08-22 19:23:54,940.940 INFO:tasks.rados.rados.0.mira097.stdout:498: finishing write tid 1 to mira09725123-498 2014-08-22 19:23:54,940.940 INFO:tasks.rados.rados.0.mira097.stdout:498: finishing write tid 2 to mira09725123-498 2014-08-22 19:23:54,941.941 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 3 to mira09725123-497 2014-08-22 19:23:54,941.941 INFO:tasks.rados.rados.0.mira097.stdout:498: finishing write tid 3 to mira09725123-498 2014-08-22 19:23:54,941.941 INFO:tasks.rados.rados.0.mira097.stdout:498: finishing write tid 5 to mira09725123-498 2014-08-22 19:23:54,942.942 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 4 to mira09725123-497 2014-08-22 19:23:54,942.942 INFO:tasks.rados.rados.0.mira097.stdout:498: finishing write tid 6 to mira09725123-498 2014-08-22 19:23:54,942.942 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 498 v 618 (ObjNum 497 snap 0 seq_num 497) dirty exists 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 6 to mira09725123-497 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:497: finishing write tid 7 to mira09725123-497 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 497 v 762 (ObjNum 496 snap 0 seq_num 496) dirty exists 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:500: finishing write tid 1 to mira09725123-500 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:500: finishing write tid 3 to mira09725123-500 2014-08-22 19:23:55,087.087 INFO:tasks.rados.rados.0.mira097.stdout:500: finishing write tid 4 to mira09725123-500 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 500 v 765 (ObjNum 499 snap 0 seq_num 499) dirty exists 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:462: done (13 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:465: done (12 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:492: done (11 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:497: done (10 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:498: done (9 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:500: done (8 left) 2014-08-22 19:23:55,088.088 INFO:tasks.rados.rados.0.mira097.stdout:501: done (7 left) 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:502: write oid 415 current snap is 0 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:502: seq_num 501 ranges {573461=1} 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:502: writing mira09725123-415 from 573461 to 573462 tid 1 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:503: snap_create 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 8 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:495: finishing write tid 5 to mira09725123-495 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 495 v 756 (ObjNum 494 snap 0 seq_num 494) dirty exists 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout:495: done (7 left) 2014-08-22 19:23:55,089.089 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 7 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:502: finishing write tid 1 to mira09725123-415 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:502: finishing write tid 3 to mira09725123-415 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:502: finishing write tid 4 to mira09725123-415 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 415 v 624 (ObjNum 501 snap 0 seq_num 501) dirty exists 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:502: done (6 left) 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:494: finishing write tid 1 to mira09725123-494 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:493: finishing write tid 7 to mira09725123-493 2014-08-22 19:23:55,090.090 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 493 v 663 (ObjNum 492 snap 0 seq_num 492) dirty exists 2014-08-22 19:23:55,091.091 INFO:tasks.rados.rados.0.mira097.stdout:493: done (5 left) 2014-08-22 19:23:55,091.091 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:23:55,091.091 INFO:tasks.rados.rados.0.mira097.stdout:494: finishing write tid 2 to mira09725123-494 2014-08-22 19:23:55,093.093 INFO:tasks.rados.rados.0.mira097.stdout:494: finishing write tid 4 to mira09725123-494 2014-08-22 19:23:55,093.093 INFO:tasks.rados.rados.0.mira097.stdout:499: finishing write tid 1 to mira09725123-499 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:499: finishing write tid 2 to mira09725123-499 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:499: finishing write tid 3 to mira09725123-499 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:499: finishing write tid 5 to mira09725123-499 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:496: finishing write tid 1 to mira09725123-496 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:496: finishing write tid 3 to mira09725123-496 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:494: finishing write tid 5 to mira09725123-494 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 494 v 672 (ObjNum 493 snap 0 seq_num 493) dirty exists 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:494: done (4 left) 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:23:55,094.094 INFO:tasks.rados.rados.0.mira097.stdout:499: finishing write tid 6 to mira09725123-499 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 499 v 678 (ObjNum 498 snap 0 seq_num 498) dirty exists 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:499: done (3 left) 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:490: finishing write tid 1 to mira09725123-490 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:490: finishing write tid 2 to mira09725123-490 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:490: finishing write tid 4 to mira09725123-490 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:489: finishing write tid 1 to mira09725123-489 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:489: finishing write tid 2 to mira09725123-489 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:489: finishing write tid 4 to mira09725123-489 2014-08-22 19:23:55,095.095 INFO:tasks.rados.rados.0.mira097.stdout:496: finishing write tid 4 to mira09725123-496 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 496 v 680 (ObjNum 495 snap 0 seq_num 495) dirty exists 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:496: done (2 left) 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:490: finishing write tid 5 to mira09725123-490 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 490 v 711 (ObjNum 489 snap 0 seq_num 489) dirty exists 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:490: done (1 left) 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:489: finishing write tid 5 to mira09725123-489 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 489 v 717 (ObjNum 488 snap 0 seq_num 488) dirty exists 2014-08-22 19:23:55,096.096 INFO:tasks.rados.rados.0.mira097.stdout:489: done (0 left) 2014-08-22 19:23:55,361.361 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.271998 7f6500d26700 1 librados: starting msgr at :/0 2014-08-22 19:23:55,361.361 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.272009 7f6500d26700 1 librados: starting objecter 2014-08-22 19:23:55,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.272278 7f6500d26700 1 librados: setting wanted keys 2014-08-22 19:23:55,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.272292 7f6500d26700 1 librados: calling monclient init 2014-08-22 19:23:55,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.274104 7f64f9d8a700 10 client.?.objecter ms_handle_connect 0x7f64fc0355e0 2014-08-22 19:23:55,363.363 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.274193 7f64f9d8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:23:55,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.276857 7f6500d26700 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:23:55,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.277054 7f6500d26700 1 librados: init done 2014-08-22 19:23:55,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.277856 7f64f9d8a700 10 client.4143.objecter ms_dispatch 0x7f64fc000f80 osd_map(25..25 src has 1..25) v3 2014-08-22 19:23:55,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.277871 7f64f9d8a700 3 client.4143.objecter handle_osd_map got epochs [25,25] > 0 2014-08-22 19:23:55,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.277874 7f64f9d8a700 3 client.4143.objecter handle_osd_map decoding full epoch 25 2014-08-22 19:23:55,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.277999 7f64f9d8a700 20 client.4143.objecter dump_active .. 0 homeless 2014-08-22 19:23:55,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.278052 7f64f9d8a700 10 client.4143.objecter ms_dispatch 0x7f64fc000f80 osd_map(25..25 src has 1..25) v3 2014-08-22 19:23:55,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.278057 7f64f9d8a700 3 client.4143.objecter handle_osd_map ignoring epochs [25,25] <= 25 2014-08-22 19:23:55,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.278059 7f64f9d8a700 20 client.4143.objecter dump_active .. 0 homeless 2014-08-22 19:23:55,459.459 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:23:55,717.717 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.374537 7f6500d26700 20 client.4143.objecter shutdown clearing up homeless session... 2014-08-22 19:23:55,717.717 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:48.375515 7f6500d26700 1 librados: shutdown 2014-08-22 19:23:55,720.720 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:23:56,288.288 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.198759 7f0431fac700 1 librados: starting msgr at :/0 2014-08-22 19:23:56,289.289 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.198769 7f0431fac700 1 librados: starting objecter 2014-08-22 19:23:56,289.289 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.199060 7f0431fac700 1 librados: setting wanted keys 2014-08-22 19:23:56,289.289 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.199075 7f0431fac700 1 librados: calling monclient init 2014-08-22 19:23:56,289.289 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.200742 7f042affd700 10 client.?.objecter ms_handle_connect 0x7f042c0355e0 2014-08-22 19:23:56,289.289 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.200876 7f042affd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:23:56,411.411 INFO:tasks.rados.rados.0.mira097.stdout:503: done (0 left) 2014-08-22 19:23:56,411.411 INFO:tasks.rados.rados.0.mira097.stdout:504: snap_create 2014-08-22 19:23:56,418.418 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.328565 7f0431fac700 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:23:56,418.418 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.328701 7f0431fac700 1 librados: init done 2014-08-22 19:23:56,418.418 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329046 7f042affd700 10 client.4145.objecter ms_dispatch 0x7f042c000f80 osd_map(26..26 src has 1..26) v3 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329059 7f042affd700 3 client.4145.objecter handle_osd_map got epochs [26,26] > 0 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329063 7f042affd700 3 client.4145.objecter handle_osd_map decoding full epoch 26 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329182 7f042affd700 20 client.4145.objecter dump_active .. 0 homeless 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329230 7f042affd700 10 client.4145.objecter ms_dispatch 0x7f042c000f80 osd_map(26..26 src has 1..26) v3 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329234 7f042affd700 3 client.4145.objecter handle_osd_map ignoring epochs [26,26] <= 26 2014-08-22 19:23:56,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.329236 7f042affd700 20 client.4145.objecter dump_active .. 0 homeless 2014-08-22 19:23:56,541.541 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.452762 7f0431fac700 20 client.4145.objecter shutdown clearing up homeless session... 2014-08-22 19:23:56,542.542 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:49.453869 7f0431fac700 1 librados: shutdown 2014-08-22 19:23:56,551.551 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003786', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001834', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:23:35.666841', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 874087720, u'health': u'HEALTH_OK', u'kb_used': 22693332, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:23:36.468081', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 877203152, u'health': u'HEALTH_OK', u'kb_used': 19577900, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:23:36.072261', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 874087712, u'health': u'HEALTH_OK', u'kb_used': 22693340, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'mon.c has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 4998582538240, u'num_pgs': 32, u'data_bytes': 1137979440, u'bytes_used': 2832494592, u'op_per_sec': 86, u'version': 51, u'pgs_by_state': [{u'count': 32, u'state_name': u'active+clean'}], u'read_bytes_sec': 13055, u'write_bytes_sec': 18929579, u'bytes_avail': 4995750043648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 26, u'num_in_osds': 5}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:23:56,553.553 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:23:56,553.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:23:56,553.553 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.005771 2014-08-22 19:23:56,554.554 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.00577074945186' 2014-08-22 19:23:57,197.197 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.107870 7fe03c4fc700 1 librados: starting msgr at :/0 2014-08-22 19:23:57,197.197 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.107878 7fe03c4fc700 1 librados: starting objecter 2014-08-22 19:23:57,197.197 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.108111 7fe03c4fc700 1 librados: setting wanted keys 2014-08-22 19:23:57,197.197 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.108120 7fe03c4fc700 1 librados: calling monclient init 2014-08-22 19:23:57,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.110372 7fe0395a9700 10 client.?.objecter ms_handle_connect 0x7fe0340355e0 2014-08-22 19:23:57,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.110430 7fe0395a9700 10 client.?.objecter resend_mon_ops 2014-08-22 19:23:57,203.203 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.114416 7fe03c4fc700 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:23:57,204.204 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.114633 7fe03c4fc700 1 librados: init done 2014-08-22 19:23:57,204.204 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115459 7fe0395a9700 10 client.4153.objecter ms_dispatch 0x7fe034000f80 osd_map(26..26 src has 1..26) v3 2014-08-22 19:23:57,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115470 7fe0395a9700 3 client.4153.objecter handle_osd_map got epochs [26,26] > 0 2014-08-22 19:23:57,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115476 7fe0395a9700 3 client.4153.objecter handle_osd_map decoding full epoch 26 2014-08-22 19:23:57,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115564 7fe0395a9700 20 client.4153.objecter dump_active .. 0 homeless 2014-08-22 19:23:57,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115669 7fe0395a9700 10 client.4153.objecter ms_dispatch 0x7fe034000f80 osd_map(26..26 src has 1..26) v3 2014-08-22 19:23:57,206.206 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115679 7fe0395a9700 3 client.4153.objecter handle_osd_map ignoring epochs [26,26] <= 26 2014-08-22 19:23:57,206.206 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.115682 7fe0395a9700 20 client.4153.objecter dump_active .. 0 homeless 2014-08-22 19:23:57,421.421 INFO:tasks.rados.rados.0.mira097.stdout:504: done (0 left) 2014-08-22 19:23:57,422.422 INFO:tasks.rados.rados.0.mira097.stdout:505: write oid 195 current snap is 2 2014-08-22 19:23:57,422.422 INFO:tasks.rados.rados.0.mira097.stdout:505: seq_num 502 ranges {517268=529971,1317267=1} 2014-08-22 19:23:57,452.452 INFO:tasks.rados.rados.0.mira097.stdout:505: writing mira09725123-195 from 517268 to 1047239 tid 1 2014-08-22 19:23:57,454.454 INFO:tasks.rados.rados.0.mira097.stdout:505: writing mira09725123-195 from 1317267 to 1317268 tid 2 2014-08-22 19:23:57,454.454 INFO:tasks.rados.rados.0.mira097.stdout:506: copy_from oid 25 from oid 196 current snap is 2 2014-08-22 19:23:57,454.454 INFO:tasks.rados.rados.0.mira097.stdout:507: rollback oid 359 current snap is 2 2014-08-22 19:23:57,455.455 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 359 to 1 2014-08-22 19:23:57,455.455 INFO:tasks.rados.rados.0.mira097.stdout:508: read oid 440 snap -1 2014-08-22 19:23:57,455.455 INFO:tasks.rados.rados.0.mira097.stdout:509: copy_from oid 272 from oid 201 current snap is 2 2014-08-22 19:23:57,455.455 INFO:tasks.rados.rados.0.mira097.stdout:510: snap_create 2014-08-22 19:23:57,455.455 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:23:57,458.458 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.00577075 (83782) 2014-08-22 19:23:57,462.462 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.372995 7fe03c4fc700 20 client.4153.objecter shutdown clearing up homeless session... 2014-08-22 19:23:57,462.462 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:50.373271 7fe03c4fc700 1 librados: shutdown 2014-08-22 19:23:57,504.504 INFO:tasks.rados.rados.0.mira097.stdout:508: expect (ObjNum 439 snap 0 seq_num 439) 2014-08-22 19:23:57,996.996 INFO:tasks.rados.rados.0.mira097.stdout:505: finishing write tid 1 to mira09725123-195 2014-08-22 19:23:57,997.997 INFO:tasks.rados.rados.0.mira097.stdout:505: finishing write tid 2 to mira09725123-195 2014-08-22 19:23:57,997.997 INFO:tasks.rados.rados.0.mira097.stdout:505: finishing write tid 4 to mira09725123-195 2014-08-22 19:23:57,997.997 INFO:tasks.rados.rados.0.mira097.stdout:505: finishing write tid 5 to mira09725123-195 2014-08-22 19:23:57,997.997 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 195 v 629 (ObjNum 502 snap 2 seq_num 502) dirty exists 2014-08-22 19:23:57,997.997 INFO:tasks.rados.rados.0.mira097.stdout:505: done (4 left) 2014-08-22 19:23:57,998.998 INFO:tasks.rados.rados.0.mira097.stdout:508: done (3 left) 2014-08-22 19:23:57,998.998 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:23:58,484.484 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 359 v 707 (ObjNum 358 snap 0 seq_num 358) dirty exists 2014-08-22 19:23:58,484.484 INFO:tasks.rados.rados.0.mira097.stdout:507: done (2 left) 2014-08-22 19:23:58,485.485 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:23:58,568.568 INFO:tasks.rados.rados.0.mira097.stdout:506: finishing copy_from to mira09725123-25 2014-08-22 19:23:58,568.568 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 25 v 260 (ObjNum 195 snap 0 seq_num 195) dirty exists 2014-08-22 19:23:58,569.569 INFO:tasks.rados.rados.0.mira097.stdout:506: finishing copy_from racing read to mira09725123-25 2014-08-22 19:23:58,569.569 INFO:tasks.rados.rados.0.mira097.stdout:506: done (1 left) 2014-08-22 19:23:58,569.569 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:23:58,575.575 INFO:tasks.rados.rados.0.mira097.stdout:509: finishing copy_from to mira09725123-272 2014-08-22 19:23:58,576.576 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 272 v 238 (ObjNum 200 snap 0 seq_num 200) dirty exists 2014-08-22 19:23:59,109.109 INFO:tasks.rados.rados.0.mira097.stdout:509: finishing copy_from racing read to mira09725123-272 2014-08-22 19:23:59,109.109 INFO:tasks.rados.rados.0.mira097.stdout:509: done (0 left) 2014-08-22 19:23:59,442.442 INFO:tasks.rados.rados.0.mira097.stdout:510: done (0 left) 2014-08-22 19:23:59,443.443 INFO:tasks.rados.rados.0.mira097.stdout:511: write oid 211 current snap is 3 2014-08-22 19:23:59,443.443 INFO:tasks.rados.rados.0.mira097.stdout:511: seq_num 503 ranges {462097=653199,1583950=418624,2062096=1} 2014-08-22 19:23:59,468.468 INFO:tasks.rados.rados.0.mira097.stdout:511: writing mira09725123-211 from 462097 to 1115296 tid 1 2014-08-22 19:23:59,483.483 INFO:tasks.rados.rados.0.mira097.stdout:511: writing mira09725123-211 from 1583950 to 2002574 tid 2 2014-08-22 19:23:59,486.486 INFO:tasks.rados.rados.0.mira097.stdout:511: writing mira09725123-211 from 2062096 to 2062097 tid 3 2014-08-22 19:23:59,487.487 INFO:tasks.rados.rados.0.mira097.stdout:512: write oid 428 current snap is 3 2014-08-22 19:23:59,487.487 INFO:tasks.rados.rados.0.mira097.stdout:512: seq_num 504 ranges {406927=626028,1547683=640243,2680962=125965} 2014-08-22 19:23:59,505.505 INFO:tasks.rados.rados.0.mira097.stdout:512: writing mira09725123-428 from 406927 to 1032955 tid 1 2014-08-22 19:23:59,526.526 INFO:tasks.rados.rados.0.mira097.stdout:512: writing mira09725123-428 from 1547683 to 2187926 tid 2 2014-08-22 19:23:59,533.533 INFO:tasks.rados.rados.0.mira097.stdout:512: writing mira09725123-428 from 2680962 to 2806927 tid 3 2014-08-22 19:23:59,534.534 INFO:tasks.rados.rados.0.mira097.stdout:513: read oid 297 snap -1 2014-08-22 19:23:59,534.534 INFO:tasks.rados.rados.0.mira097.stdout:511: finishing write tid 1 to mira09725123-211 2014-08-22 19:23:59,534.534 INFO:tasks.rados.rados.0.mira097.stdout:511: finishing write tid 2 to mira09725123-211 2014-08-22 19:23:59,534.534 INFO:tasks.rados.rados.0.mira097.stdout:511: finishing write tid 3 to mira09725123-211 2014-08-22 19:23:59,534.534 INFO:tasks.rados.rados.0.mira097.stdout:511: finishing write tid 5 to mira09725123-211 2014-08-22 19:23:59,535.535 INFO:tasks.rados.rados.0.mira097.stdout:511: finishing write tid 6 to mira09725123-211 2014-08-22 19:23:59,535.535 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 211 v 756 (ObjNum 503 snap 3 seq_num 503) dirty exists 2014-08-22 19:23:59,535.535 INFO:tasks.rados.rados.0.mira097.stdout:512: finishing write tid 1 to mira09725123-428 2014-08-22 19:23:59,535.535 INFO:tasks.rados.rados.0.mira097.stdout:511: done (2 left) 2014-08-22 19:23:59,536.536 INFO:tasks.rados.rados.0.mira097.stdout:514: write oid 293 current snap is 3 2014-08-22 19:23:59,536.536 INFO:tasks.rados.rados.0.mira097.stdout:514: seq_num 505 ranges {618509=750280,1893495=594086,3018508=1} 2014-08-22 19:23:59,615.615 INFO:tasks.rados.rados.0.mira097.stdout:514: writing mira09725123-293 from 618509 to 1368789 tid 1 2014-08-22 19:23:59,615.615 INFO:tasks.rados.rados.0.mira097.stdout:514: writing mira09725123-293 from 1893495 to 2487581 tid 2 2014-08-22 19:23:59,615.615 INFO:tasks.rados.rados.0.mira097.stdout:514: writing mira09725123-293 from 3018508 to 3018509 tid 3 2014-08-22 19:23:59,615.615 INFO:tasks.rados.rados.0.mira097.stdout:515: write oid 340 current snap is 3 2014-08-22 19:23:59,615.615 INFO:tasks.rados.rados.0.mira097.stdout:515: seq_num 506 ranges {563339=605732,1739851=664282,2815637=638815,3763338=1} 2014-08-22 19:23:59,616.616 INFO:tasks.rados.rados.0.mira097.stdout:515: writing mira09725123-340 from 563339 to 1169071 tid 1 2014-08-22 19:23:59,631.631 INFO:tasks.rados.rados.0.mira097.stdout:515: writing mira09725123-340 from 1739851 to 2404133 tid 2 2014-08-22 19:23:59,653.653 INFO:tasks.rados.rados.0.mira097.stdout:515: writing mira09725123-340 from 2815637 to 3454452 tid 3 2014-08-22 19:23:59,658.658 INFO:tasks.rados.rados.0.mira097.stdout:515: writing mira09725123-340 from 3763338 to 3763339 tid 4 2014-08-22 19:23:59,659.659 INFO:tasks.rados.rados.0.mira097.stdout:516: read oid 227 snap -1 2014-08-22 19:23:59,659.659 INFO:tasks.rados.rados.0.mira097.stdout:513: expect (ObjNum 296 snap 0 seq_num 296) 2014-08-22 19:23:59,725.725 INFO:tasks.rados.rados.0.mira097.stdout:513: done (4 left) 2014-08-22 19:23:59,725.725 INFO:tasks.rados.rados.0.mira097.stdout:517: rollback oid 309 current snap is 3 2014-08-22 19:23:59,725.725 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 309 to 2 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:512: finishing write tid 2 to mira09725123-428 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:512: finishing write tid 3 to mira09725123-428 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:512: finishing write tid 5 to mira09725123-428 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:512: finishing write tid 6 to mira09725123-428 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 428 v 765 (ObjNum 504 snap 3 seq_num 504) dirty exists 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 1 to mira09725123-340 2014-08-22 19:23:59,726.726 INFO:tasks.rados.rados.0.mira097.stdout:514: finishing write tid 1 to mira09725123-293 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:514: finishing write tid 2 to mira09725123-293 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:514: finishing write tid 3 to mira09725123-293 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:514: finishing write tid 5 to mira09725123-293 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:512: done (4 left) 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:518: write oid 115 current snap is 3 2014-08-22 19:23:59,727.727 INFO:tasks.rados.rados.0.mira097.stdout:518: seq_num 507 ranges {506119=1} 2014-08-22 19:23:59,731.731 INFO:tasks.rados.rados.0.mira097.stdout:518: writing mira09725123-115 from 506119 to 506120 tid 1 2014-08-22 19:23:59,731.731 INFO:tasks.rados.rados.0.mira097.stdout:519: delete oid 145 current snap is 3 2014-08-22 19:23:59,873.873 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 2 to mira09725123-340 2014-08-22 19:23:59,898.898 INFO:tasks.rados.rados.0.mira097.stdout:514: finishing write tid 6 to mira09725123-293 2014-08-22 19:23:59,898.898 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 293 v 777 (ObjNum 505 snap 3 seq_num 505) dirty exists 2014-08-22 19:23:59,920.920 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 3 to mira09725123-340 2014-08-22 19:23:59,923.923 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 4 to mira09725123-340 2014-08-22 19:23:59,924.924 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 6 to mira09725123-340 2014-08-22 19:23:59,924.924 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 309 v 767 (ObjNum 308 snap 0 seq_num 308) dirty exists 2014-08-22 19:23:59,924.924 INFO:tasks.rados.rados.0.mira097.stdout:518: finishing write tid 1 to mira09725123-115 2014-08-22 19:23:59,926.926 INFO:tasks.rados.rados.0.mira097.stdout:518: finishing write tid 3 to mira09725123-115 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:518: finishing write tid 4 to mira09725123-115 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 115 v 640 (ObjNum 507 snap 3 seq_num 507) dirty exists 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:514: done (5 left) 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:517: done (4 left) 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:518: done (3 left) 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:519: done (2 left) 2014-08-22 19:23:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:520: delete oid 411 current snap is 3 2014-08-22 19:24:00,018.018 INFO:tasks.rados.rados.0.mira097.stdout:520: done (2 left) 2014-08-22 19:24:00,019.019 INFO:tasks.rados.rados.0.mira097.stdout:521: read oid 318 snap 0 2014-08-22 19:24:00,019.019 INFO:tasks.rados.rados.0.mira097.stdout:522: snap_create 2014-08-22 19:24:00,019.019 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:24:00,046.046 INFO:tasks.rados.rados.0.mira097.stdout:515: finishing write tid 7 to mira09725123-340 2014-08-22 19:24:00,046.046 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 340 v 720 (ObjNum 506 snap 3 seq_num 506) dirty exists 2014-08-22 19:24:00,046.046 INFO:tasks.rados.rados.0.mira097.stdout:515: done (2 left) 2014-08-22 19:24:00,048.048 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:00,121.121 INFO:tasks.rados.rados.0.mira097.stdout:516: expect (ObjNum 226 snap 0 seq_num 226) 2014-08-22 19:24:01,158.158 INFO:tasks.rados.rados.0.mira097.stdout:521: expect (ObjNum 317 snap 0 seq_num 317) 2014-08-22 19:24:02,467.467 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 0] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:02,468.468 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:02,468.468 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:24:02,469.469 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:02,743.743 INFO:tasks.rados.rados.0.mira097.stdout:516: done (1 left) 2014-08-22 19:24:02,743.743 INFO:tasks.rados.rados.0.mira097.stdout:521: done (0 left) 2014-08-22 19:24:02,950.950 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.861665 7f0989ab4700 1 librados: starting msgr at :/0 2014-08-22 19:24:02,951.951 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.861674 7f0989ab4700 1 librados: starting objecter 2014-08-22 19:24:02,951.951 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.861986 7f0989ab4700 1 librados: setting wanted keys 2014-08-22 19:24:02,951.951 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.861995 7f0989ab4700 1 librados: calling monclient init 2014-08-22 19:24:02,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.864247 7f09827fc700 10 client.?.objecter ms_handle_connect 0x7f09840355e0 2014-08-22 19:24:02,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.864313 7f09827fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:02,957.957 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.867979 7f0989ab4700 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:02,957.957 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.868162 7f0989ab4700 1 librados: init done 2014-08-22 19:24:02,958.958 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869036 7f09827fc700 10 client.4156.objecter ms_dispatch 0x7f0984000f80 osd_map(29..29 src has 1..29) v3 2014-08-22 19:24:02,958.958 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869049 7f09827fc700 3 client.4156.objecter handle_osd_map got epochs [29,29] > 0 2014-08-22 19:24:02,958.958 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869052 7f09827fc700 3 client.4156.objecter handle_osd_map decoding full epoch 29 2014-08-22 19:24:03,019.019 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869226 7f09827fc700 20 client.4156.objecter dump_active .. 0 homeless 2014-08-22 19:24:03,019.019 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869293 7f09827fc700 10 client.4156.objecter ms_dispatch 0x7f0984000f80 osd_map(29..29 src has 1..29) v3 2014-08-22 19:24:03,019.019 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869298 7f09827fc700 3 client.4156.objecter handle_osd_map ignoring epochs [29,29] <= 29 2014-08-22 19:24:03,019.019 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.869301 7f09827fc700 20 client.4156.objecter dump_active .. 0 homeless 2014-08-22 19:24:03,134.134 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:03,311.311 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.965842 7f0989ab4700 20 client.4156.objecter shutdown clearing up homeless session... 2014-08-22 19:24:03,312.312 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:55.966920 7f0989ab4700 1 librados: shutdown 2014-08-22 19:24:03,314.314 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 24' 2014-08-22 19:24:03,459.459 INFO:tasks.rados.rados.0.mira097.stdout:522: done (0 left) 2014-08-22 19:24:03,459.459 INFO:tasks.rados.rados.0.mira097.stdout:523: snap_create 2014-08-22 19:24:03,904.904 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.815380 7f8b221eb700 1 librados: starting msgr at :/0 2014-08-22 19:24:03,904.904 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.815394 7f8b221eb700 1 librados: starting objecter 2014-08-22 19:24:03,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.815634 7f8b221eb700 1 librados: setting wanted keys 2014-08-22 19:24:03,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.815644 7f8b221eb700 1 librados: calling monclient init 2014-08-22 19:24:03,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.817390 7f8b1affd700 10 client.?.objecter ms_handle_connect 0x7f8b1c0355e0 2014-08-22 19:24:03,906.906 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.817470 7f8b1affd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:03,910.910 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.820763 7f8b221eb700 10 client.4146.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:03,910.910 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.820945 7f8b221eb700 1 librados: init done 2014-08-22 19:24:03,910.910 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821540 7f8b1affd700 10 client.4146.objecter ms_dispatch 0x7f8b1c000f80 osd_map(30..30 src has 1..30) v3 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821564 7f8b1affd700 3 client.4146.objecter handle_osd_map got epochs [30,30] > 0 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821571 7f8b1affd700 3 client.4146.objecter handle_osd_map decoding full epoch 30 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821715 7f8b1affd700 20 client.4146.objecter dump_active .. 0 homeless 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821777 7f8b1affd700 10 client.4146.objecter ms_dispatch 0x7f8b1c000f80 osd_map(30..30 src has 1..30) v3 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821784 7f8b1affd700 3 client.4146.objecter handle_osd_map ignoring epochs [30,30] <= 30 2014-08-22 19:24:03,911.911 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:56.821788 7f8b1affd700 20 client.4146.objecter dump_active .. 0 homeless 2014-08-22 19:24:04,466.466 INFO:tasks.rados.rados.0.mira097.stdout:523: done (0 left) 2014-08-22 19:24:04,467.467 INFO:tasks.rados.rados.0.mira097.stdout:524: rollback oid 251 current snap is 5 2014-08-22 19:24:04,467.467 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 251 to 1 2014-08-22 19:24:04,467.467 INFO:tasks.rados.rados.0.mira097.stdout:525: snap_remove snap 4 2014-08-22 19:24:04,467.467 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 24 2014-08-22 19:24:04,471.471 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:57.381438 7f8b221eb700 20 client.4146.objecter shutdown clearing up homeless session... 2014-08-22 19:24:04,471.471 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:23:57.382113 7f8b221eb700 1 librados: shutdown 2014-08-22 19:24:04,505.505 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 251 v 729 (ObjNum 250 snap 0 seq_num 250) dirty exists 2014-08-22 19:24:05,474.474 INFO:tasks.rados.rados.0.mira097.stdout:524: done (1 left) 2014-08-22 19:24:05,474.474 INFO:tasks.rados.rados.0.mira097.stdout:525: done (0 left) 2014-08-22 19:24:05,475.475 INFO:tasks.rados.rados.0.mira097.stdout:526: read oid 337 snap 3 2014-08-22 19:24:05,481.481 INFO:tasks.rados.rados.0.mira097.stdout:527: read oid 257 snap -1 2014-08-22 19:24:05,481.481 INFO:tasks.rados.rados.0.mira097.stdout:528: read oid 423 snap -1 2014-08-22 19:24:05,482.482 INFO:tasks.rados.rados.0.mira097.stdout:529: write oid 302 current snap is 5 2014-08-22 19:24:05,482.482 INFO:tasks.rados.rados.0.mira097.stdout:529: seq_num 510 ranges {607362=680470,1724188=447910,2207361=1} 2014-08-22 19:24:05,505.505 INFO:tasks.rados.rados.0.mira097.stdout:529: writing mira09725123-302 from 607362 to 1287832 tid 1 2014-08-22 19:24:05,523.523 INFO:tasks.rados.rados.0.mira097.stdout:529: writing mira09725123-302 from 1724188 to 2172098 tid 2 2014-08-22 19:24:05,524.524 INFO:tasks.rados.rados.0.mira097.stdout:529: writing mira09725123-302 from 2207361 to 2207362 tid 3 2014-08-22 19:24:05,524.524 INFO:tasks.rados.rados.0.mira097.stdout:530: snap_remove snap 2 2014-08-22 19:24:05,524.524 INFO:tasks.rados.rados.0.mira097.stdout:527: expect (ObjNum 256 snap 0 seq_num 256) 2014-08-22 19:24:05,704.704 INFO:tasks.rados.rados.0.mira097.stdout:526: expect (ObjNum 336 snap 0 seq_num 336) 2014-08-22 19:24:05,741.741 INFO:tasks.rados.rados.0.mira097.stdout:528: expect (ObjNum 422 snap 0 seq_num 422) 2014-08-22 19:24:06,741.741 INFO:tasks.rados.rados.0.mira097.stdout:529: finishing write tid 1 to mira09725123-302 2014-08-22 19:24:06,741.741 INFO:tasks.rados.rados.0.mira097.stdout:529: finishing write tid 2 to mira09725123-302 2014-08-22 19:24:06,741.741 INFO:tasks.rados.rados.0.mira097.stdout:529: finishing write tid 3 to mira09725123-302 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:529: finishing write tid 5 to mira09725123-302 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:529: finishing write tid 6 to mira09725123-302 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 302 v 772 (ObjNum 510 snap 5 seq_num 510) dirty exists 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:526: done (4 left) 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:527: done (3 left) 2014-08-22 19:24:06,742.742 INFO:tasks.rados.rados.0.mira097.stdout:528: done (2 left) 2014-08-22 19:24:06,743.743 INFO:tasks.rados.rados.0.mira097.stdout:529: done (1 left) 2014-08-22 19:24:06,743.743 INFO:tasks.rados.rados.0.mira097.stdout:530: done (0 left) 2014-08-22 19:24:06,743.743 INFO:tasks.rados.rados.0.mira097.stdout:531: delete oid 495 current snap is 5 2014-08-22 19:24:06,748.748 INFO:tasks.rados.rados.0.mira097.stdout:531: done (0 left) 2014-08-22 19:24:06,748.748 INFO:tasks.rados.rados.0.mira097.stdout:532: write oid 149 current snap is 5 2014-08-22 19:24:06,748.748 INFO:tasks.rados.rados.0.mira097.stdout:532: seq_num 512 ranges {647421=776527,1917380=587277,2647420=1} 2014-08-22 19:24:06,780.780 INFO:tasks.rados.rados.0.mira097.stdout:532: writing mira09725123-149 from 647421 to 1423948 tid 1 2014-08-22 19:24:06,801.801 INFO:tasks.rados.rados.0.mira097.stdout:532: writing mira09725123-149 from 1917380 to 2504657 tid 2 2014-08-22 19:24:06,803.803 INFO:tasks.rados.rados.0.mira097.stdout:532: writing mira09725123-149 from 2647420 to 2647421 tid 3 2014-08-22 19:24:06,803.803 INFO:tasks.rados.rados.0.mira097.stdout:533: copy_from oid 198 from oid 208 current snap is 5 2014-08-22 19:24:06,803.803 INFO:tasks.rados.rados.0.mira097.stdout:532: finishing write tid 1 to mira09725123-149 2014-08-22 19:24:06,915.915 INFO:tasks.rados.rados.0.mira097.stdout:534: delete oid 263 current snap is 5 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:534: done (2 left) 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:535: copy_from oid 243 from oid 410 current snap is 5 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:536: read oid 426 snap 1 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:537: write oid 184 current snap is 5 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:537: seq_num 514 ranges {404857=603783,1441772=762739,2618719=603923,3604856=1} 2014-08-22 19:24:06,916.916 INFO:tasks.rados.rados.0.mira097.stdout:537: writing mira09725123-184 from 404857 to 1008640 tid 1 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:537: writing mira09725123-184 from 1441772 to 2204511 tid 2 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:537: writing mira09725123-184 from 2618719 to 3222642 tid 3 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:537: writing mira09725123-184 from 3604856 to 3604857 tid 4 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:538: write oid 77 current snap is 5 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:538: seq_num 515 ranges {500086=727012,1822656=584358,3121768=178318} 2014-08-22 19:24:06,917.917 INFO:tasks.rados.rados.0.mira097.stdout:538: writing mira09725123-77 from 500086 to 1227098 tid 1 2014-08-22 19:24:06,936.936 INFO:tasks.rados.rados.0.mira097.stdout:538: writing mira09725123-77 from 1822656 to 2407014 tid 2 2014-08-22 19:24:06,946.946 INFO:tasks.rados.rados.0.mira097.stdout:538: writing mira09725123-77 from 3121768 to 3300086 tid 3 2014-08-22 19:24:06,947.947 INFO:tasks.rados.rados.0.mira097.stdout:539: copy_from oid 111 from oid 348 current snap is 5 2014-08-22 19:24:06,947.947 INFO:tasks.rados.rados.0.mira097.stdout:532: finishing write tid 2 to mira09725123-149 2014-08-22 19:24:06,947.947 INFO:tasks.rados.rados.0.mira097.stdout:532: finishing write tid 3 to mira09725123-149 2014-08-22 19:24:06,948.948 INFO:tasks.rados.rados.0.mira097.stdout:532: finishing write tid 5 to mira09725123-149 2014-08-22 19:24:06,948.948 INFO:tasks.rados.rados.0.mira097.stdout:532: finishing write tid 6 to mira09725123-149 2014-08-22 19:24:06,948.948 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 149 v 659 (ObjNum 512 snap 5 seq_num 512) dirty exists 2014-08-22 19:24:06,948.948 INFO:tasks.rados.rados.0.mira097.stdout:536: expect (ObjNum 425 snap 0 seq_num 425) 2014-08-22 19:24:07,001.001 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 1 to mira09725123-184 2014-08-22 19:24:07,001.001 INFO:tasks.rados.rados.0.mira097.stdout:533: finishing copy_from to mira09725123-198 2014-08-22 19:24:07,001.001 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 198 v 241 (ObjNum 207 snap 0 seq_num 207) dirty exists 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:533: finishing copy_from racing read to mira09725123-198 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 2 to mira09725123-184 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:532: done (6 left) 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:533: done (5 left) 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:536: done (4 left) 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:540: read oid 495 snap -1 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:541: write oid 300 current snap is 5 2014-08-22 19:24:07,002.002 INFO:tasks.rados.rados.0.mira097.stdout:541: seq_num 516 ranges {692468=400000} 2014-08-22 19:24:07,022.022 INFO:tasks.rados.rados.0.mira097.stdout:541: writing mira09725123-300 from 692468 to 1092468 tid 1 2014-08-22 19:24:07,022.022 INFO:tasks.rados.rados.0.mira097.stdout:542: rollback oid 421 current snap is 5 2014-08-22 19:24:07,022.022 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 421 to 3 2014-08-22 19:24:07,023.023 INFO:tasks.rados.rados.0.mira097.stdout:540: done (6 left) 2014-08-22 19:24:07,023.023 INFO:tasks.rados.rados.0.mira097.stdout:543: delete oid 358 current snap is 5 2014-08-22 19:24:07,043.043 INFO:tasks.rados.rados.0.mira097.stdout:538: finishing write tid 1 to mira09725123-77 2014-08-22 19:24:07,044.044 INFO:tasks.rados.rados.0.mira097.stdout:538: finishing write tid 2 to mira09725123-77 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:538: finishing write tid 3 to mira09725123-77 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:538: finishing write tid 5 to mira09725123-77 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:538: finishing write tid 6 to mira09725123-77 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 77 v 668 (ObjNum 515 snap 5 seq_num 515) dirty exists 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 3 to mira09725123-184 2014-08-22 19:24:07,093.093 INFO:tasks.rados.rados.0.mira097.stdout:535: finishing copy_from to mira09725123-243 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 243 v 549 (ObjNum 409 snap 0 seq_num 409) dirty exists 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:538: done (6 left) 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:543: done (5 left) 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:544: write oid 256 current snap is 5 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:544: seq_num 518 ranges {483950=1} 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:544: writing mira09725123-256 from 483950 to 483951 tid 1 2014-08-22 19:24:07,094.094 INFO:tasks.rados.rados.0.mira097.stdout:545: rollback oid 124 current snap is 5 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 124 to 1 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 4 to mira09725123-184 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 6 to mira09725123-184 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:546: snap_create 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 7 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:537: finishing write tid 7 to mira09725123-184 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 184 v 793 (ObjNum 514 snap 5 seq_num 514) dirty exists 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:539: finishing copy_from to mira09725123-111 2014-08-22 19:24:07,095.095 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 111 v 436 (ObjNum 347 snap 0 seq_num 347) dirty exists 2014-08-22 19:24:07,096.096 INFO:tasks.rados.rados.0.mira097.stdout:537: done (6 left) 2014-08-22 19:24:07,096.096 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:24:07,227.227 INFO:tasks.rados.rados.0.mira097.stdout:535: finishing copy_from racing read to mira09725123-243 2014-08-22 19:24:07,227.227 INFO:tasks.rados.rados.0.mira097.stdout:535: done (5 left) 2014-08-22 19:24:07,227.227 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:24:07,229.229 INFO:tasks.rados.rados.0.mira097.stdout:541: finishing write tid 1 to mira09725123-300 2014-08-22 19:24:07,229.229 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 421 v 809 (ObjNum 420 snap 0 seq_num 420) dirty exists 2014-08-22 19:24:07,230.230 INFO:tasks.rados.rados.0.mira097.stdout:542: done (4 left) 2014-08-22 19:24:07,230.230 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:24:07,230.230 INFO:tasks.rados.rados.0.mira097.stdout:541: finishing write tid 3 to mira09725123-300 2014-08-22 19:24:07,231.231 INFO:tasks.rados.rados.0.mira097.stdout:544: finishing write tid 1 to mira09725123-256 2014-08-22 19:24:07,231.231 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 124 v 820 (ObjNum 123 snap 0 seq_num 123) dirty exists 2014-08-22 19:24:07,232.232 INFO:tasks.rados.rados.0.mira097.stdout:545: done (3 left) 2014-08-22 19:24:07,232.232 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:24:07,232.232 INFO:tasks.rados.rados.0.mira097.stdout:544: finishing write tid 3 to mira09725123-256 2014-08-22 19:24:07,232.232 INFO:tasks.rados.rados.0.mira097.stdout:544: finishing write tid 4 to mira09725123-256 2014-08-22 19:24:07,233.233 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 256 v 672 (ObjNum 518 snap 5 seq_num 518) dirty exists 2014-08-22 19:24:07,233.233 INFO:tasks.rados.rados.0.mira097.stdout:544: done (2 left) 2014-08-22 19:24:07,233.233 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:07,263.263 INFO:tasks.rados.rados.0.mira097.stdout:539: finishing copy_from racing read to mira09725123-111 2014-08-22 19:24:07,263.263 INFO:tasks.rados.rados.0.mira097.stdout:539: done (1 left) 2014-08-22 19:24:07,263.263 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:07,407.407 INFO:tasks.rados.rados.0.mira097.stdout:541: finishing write tid 4 to mira09725123-300 2014-08-22 19:24:07,407.407 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 300 v 804 (ObjNum 516 snap 5 seq_num 516) dirty exists 2014-08-22 19:24:07,407.407 INFO:tasks.rados.rados.0.mira097.stdout:541: done (0 left) 2014-08-22 19:24:07,516.516 INFO:tasks.rados.rados.0.mira097.stdout:546: done (0 left) 2014-08-22 19:24:07,516.516 INFO:tasks.rados.rados.0.mira097.stdout:547: read oid 358 snap -1 2014-08-22 19:24:07,516.516 INFO:tasks.rados.rados.0.mira097.stdout:548: copy_from oid 42 from oid 9 current snap is 6 2014-08-22 19:24:07,516.516 INFO:tasks.rados.rados.0.mira097.stdout:549: snap_create 2014-08-22 19:24:07,517.517 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:07,521.521 INFO:tasks.rados.rados.0.mira097.stdout:547: done (1 left) 2014-08-22 19:24:07,521.521 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:07,635.635 INFO:tasks.rados.rados.0.mira097.stdout:548: finishing copy_from to mira09725123-42 2014-08-22 19:24:07,635.635 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 42 v 8 (ObjNum 8 snap 0 seq_num 8) dirty exists 2014-08-22 19:24:07,639.639 INFO:tasks.rados.rados.0.mira097.stdout:548: finishing copy_from racing read to mira09725123-42 2014-08-22 19:24:07,639.639 INFO:tasks.rados.rados.0.mira097.stdout:548: done (0 left) 2014-08-22 19:24:08,526.526 INFO:tasks.rados.rados.0.mira097.stdout:549: done (0 left) 2014-08-22 19:24:08,527.527 INFO:tasks.rados.rados.0.mira097.stdout:550: snap_create 2014-08-22 19:24:09,477.477 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 0] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:09,477.477 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:09,477.477 INFO:tasks.thrashosds.thrasher:Adding osd 1 2014-08-22 19:24:09,478.478 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1' 2014-08-22 19:24:09,537.537 INFO:tasks.rados.rados.0.mira097.stdout:550: done (0 left) 2014-08-22 19:24:09,537.537 INFO:tasks.rados.rados.0.mira097.stdout:551: copy_from oid 431 from oid 145 current snap is 8 2014-08-22 19:24:09,537.537 INFO:tasks.rados.rados.0.mira097.stdout:552: rollback oid 62 current snap is 8 2014-08-22 19:24:09,537.537 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 62 to 3 2014-08-22 19:24:09,538.538 INFO:tasks.rados.rados.0.mira097.stdout:553: write oid 18 current snap is 8 2014-08-22 19:24:09,538.538 INFO:tasks.rados.rados.0.mira097.stdout:553: seq_num 519 ranges {793709=402773,1789841=465139,2771598=22111} 2014-08-22 19:24:09,566.566 INFO:tasks.rados.rados.0.mira097.stdout:553: writing mira09725123-18 from 793709 to 1196482 tid 1 2014-08-22 19:24:09,584.584 INFO:tasks.rados.rados.0.mira097.stdout:553: writing mira09725123-18 from 1789841 to 2254980 tid 2 2014-08-22 19:24:09,588.588 INFO:tasks.rados.rados.0.mira097.stdout:553: writing mira09725123-18 from 2771598 to 2793709 tid 3 2014-08-22 19:24:09,588.588 INFO:tasks.rados.rados.0.mira097.stdout:554: read oid 289 snap -1 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:551: finishing copy_from to mira09725123-431 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:551: got expected ENOENT (src dne) 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:551: finishing copy_from racing read to mira09725123-431 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 62 v 747 (ObjNum 61 snap 0 seq_num 61) dirty exists 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:551: done (3 left) 2014-08-22 19:24:09,589.589 INFO:tasks.rados.rados.0.mira097.stdout:552: done (2 left) 2014-08-22 19:24:09,711.711 INFO:tasks.rados.rados.0.mira097.stdout:555: write oid 311 current snap is 8 2014-08-22 19:24:09,711.711 INFO:tasks.rados.rados.0.mira097.stdout:555: seq_num 520 ranges {488939=774577,1867877=621062} 2014-08-22 19:24:09,711.711 INFO:tasks.rados.rados.0.mira097.stdout:555: writing mira09725123-311 from 488939 to 1263516 tid 1 2014-08-22 19:24:09,711.711 INFO:tasks.rados.rados.0.mira097.stdout:555: writing mira09725123-311 from 1867877 to 2488939 tid 2 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:556: delete oid 216 current snap is 8 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:555: finishing write tid 1 to mira09725123-311 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:555: finishing write tid 2 to mira09725123-311 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:555: finishing write tid 4 to mira09725123-311 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:556: done (3 left) 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:557: write oid 119 current snap is 8 2014-08-22 19:24:09,712.712 INFO:tasks.rados.rados.0.mira097.stdout:557: seq_num 522 ranges {777574=753258,1942669=578573,2973227=783559,3977573=1} 2014-08-22 19:24:09,713.713 INFO:tasks.rados.rados.0.mira097.stdout:557: writing mira09725123-119 from 777574 to 1530832 tid 1 2014-08-22 19:24:09,722.722 INFO:tasks.rados.rados.0.mira097.stdout:557: writing mira09725123-119 from 1942669 to 2521242 tid 2 2014-08-22 19:24:09,750.750 INFO:tasks.rados.rados.0.mira097.stdout:557: writing mira09725123-119 from 2973227 to 3756786 tid 3 2014-08-22 19:24:09,755.755 INFO:tasks.rados.rados.0.mira097.stdout:557: writing mira09725123-119 from 3977573 to 3977574 tid 4 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:558: rollback oid 250 current snap is 8 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 250 to 6 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:555: finishing write tid 5 to mira09725123-311 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 311 v 817 (ObjNum 520 snap 8 seq_num 520) dirty exists 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:555: done (4 left) 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:559: read oid 403 snap -1 2014-08-22 19:24:09,761.761 INFO:tasks.rados.rados.0.mira097.stdout:560: copy_from oid 75 from oid 446 current snap is 8 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:561: rollback oid 141 current snap is 8 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 141 to 6 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:562: read oid 252 snap -1 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:563: read oid 216 snap 7 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:564: rollback oid 329 current snap is 8 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 329 to 1 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:565: copy_from oid 100 from oid 459 current snap is 8 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:566: copy_from oid 339 from oid 346 current snap is 8 2014-08-22 19:24:09,762.762 INFO:tasks.rados.rados.0.mira097.stdout:567: copy_from oid 52 from oid 283 current snap is 8 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:568: read oid 364 snap 3 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:569: write oid 412 current snap is 8 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:569: seq_num 523 ranges {190179=1} 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:569: writing mira09725123-412 from 190179 to 190180 tid 1 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:570: write oid 170 current snap is 8 2014-08-22 19:24:09,763.763 INFO:tasks.rados.rados.0.mira097.stdout:570: seq_num 524 ranges {685410=697891,2135862=754035,3507902=377508} 2014-08-22 19:24:09,782.782 INFO:tasks.rados.rados.0.mira097.stdout:570: writing mira09725123-170 from 685410 to 1383301 tid 1 2014-08-22 19:24:09,810.810 INFO:tasks.rados.rados.0.mira097.stdout:570: writing mira09725123-170 from 2135862 to 2889897 tid 2 2014-08-22 19:24:09,827.827 INFO:tasks.rados.rados.0.mira097.stdout:570: writing mira09725123-170 from 3507902 to 3885410 tid 3 2014-08-22 19:24:09,828.828 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 16 2014-08-22 19:24:09,829.829 INFO:tasks.rados.rados.0.mira097.stdout:563: expect (ObjNum 215 snap 0 seq_num 215) 2014-08-22 19:24:09,973.973 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.883141 7f0bae71d700 1 librados: starting msgr at :/0 2014-08-22 19:24:09,973.973 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.883148 7f0bae71d700 1 librados: starting objecter 2014-08-22 19:24:09,973.973 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.883298 7f0bae71d700 1 librados: setting wanted keys 2014-08-22 19:24:09,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.883305 7f0bae71d700 1 librados: calling monclient init 2014-08-22 19:24:09,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.884079 7f0ba6ffd700 10 client.?.objecter ms_handle_connect 0x7f0ba80355e0 2014-08-22 19:24:09,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.884106 7f0ba6ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:09,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.885468 7f0bae71d700 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:09,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.885629 7f0bae71d700 1 librados: init done 2014-08-22 19:24:09,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886025 7f0ba6ffd700 10 client.4148.objecter ms_dispatch 0x7f0ba8000f80 osd_map(36..36 src has 1..36) v3 2014-08-22 19:24:09,976.976 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886031 7f0ba6ffd700 3 client.4148.objecter handle_osd_map got epochs [36,36] > 0 2014-08-22 19:24:09,976.976 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886032 7f0ba6ffd700 3 client.4148.objecter handle_osd_map decoding full epoch 36 2014-08-22 19:24:09,982.982 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886178 7f0ba6ffd700 20 client.4148.objecter dump_active .. 0 homeless 2014-08-22 19:24:09,982.982 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886209 7f0ba6ffd700 10 client.4148.objecter ms_dispatch 0x7f0ba8000f80 osd_map(36..36 src has 1..36) v3 2014-08-22 19:24:09,982.982 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886212 7f0ba6ffd700 3 client.4148.objecter handle_osd_map ignoring epochs [36,36] <= 36 2014-08-22 19:24:09,983.983 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:02.886215 7f0ba6ffd700 20 client.4148.objecter dump_active .. 0 homeless 2014-08-22 19:24:10,241.241 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 1 to mira09725123-119 2014-08-22 19:24:10,241.241 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 2 to mira09725123-119 2014-08-22 19:24:10,242.242 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 3 to mira09725123-119 2014-08-22 19:24:10,242.242 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 4 to mira09725123-119 2014-08-22 19:24:10,242.242 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 6 to mira09725123-119 2014-08-22 19:24:10,242.242 INFO:tasks.rados.rados.0.mira097.stdout:569: finishing write tid 1 to mira09725123-412 2014-08-22 19:24:10,242.242 INFO:tasks.rados.rados.0.mira097.stdout:563: done (15 left) 2014-08-22 19:24:10,245.245 INFO:tasks.rados.rados.0.mira097.stdout:571: write oid 85 current snap is 8 2014-08-22 19:24:10,246.246 INFO:tasks.rados.rados.0.mira097.stdout:571: seq_num 525 ranges {477792=421119,1413898=263894} 2014-08-22 19:24:10,263.263 INFO:tasks.rados.rados.0.mira097.stdout:571: writing mira09725123-85 from 477792 to 898911 tid 1 2014-08-22 19:24:10,273.273 INFO:tasks.rados.rados.0.mira097.stdout:571: writing mira09725123-85 from 1413898 to 1677792 tid 2 2014-08-22 19:24:10,276.276 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 16 2014-08-22 19:24:10,277.277 INFO:tasks.rados.rados.0.mira097.stdout:569: finishing write tid 3 to mira09725123-412 2014-08-22 19:24:10,277.277 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 329 v 767 (ObjNum 328 snap 0 seq_num 328) dirty exists 2014-08-22 19:24:10,277.277 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 141 v 774 (ObjNum 140 snap 0 seq_num 140) dirty exists 2014-08-22 19:24:10,277.277 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 250 v 776 (ObjNum 249 snap 0 seq_num 249) dirty exists 2014-08-22 19:24:10,277.277 INFO:tasks.rados.rados.0.mira097.stdout:559: expect (ObjNum 402 snap 0 seq_num 402) 2014-08-22 19:24:10,589.589 INFO:tasks.rados.rados.0.mira097.stdout:570: finishing write tid 1 to mira09725123-170 2014-08-22 19:24:10,589.589 INFO:tasks.rados.rados.0.mira097.stdout:570: finishing write tid 2 to mira09725123-170 2014-08-22 19:24:10,589.589 INFO:tasks.rados.rados.0.mira097.stdout:570: finishing write tid 3 to mira09725123-170 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:570: finishing write tid 5 to mira09725123-170 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:566: finishing copy_from to mira09725123-339 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 339 v 453 (ObjNum 345 snap 0 seq_num 345) dirty exists 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:567: finishing copy_from to mira09725123-52 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 52 v 340 (ObjNum 282 snap 0 seq_num 282) dirty exists 2014-08-22 19:24:10,590.590 INFO:tasks.rados.rados.0.mira097.stdout:560: finishing copy_from to mira09725123-75 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 75 v 540 (ObjNum 445 snap 0 seq_num 445) dirty exists 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:553: finishing write tid 1 to mira09725123-18 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:553: finishing write tid 2 to mira09725123-18 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:553: finishing write tid 3 to mira09725123-18 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:553: finishing write tid 5 to mira09725123-18 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:553: finishing write tid 6 to mira09725123-18 2014-08-22 19:24:10,591.591 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 18 v 691 (ObjNum 519 snap 8 seq_num 519) dirty exists 2014-08-22 19:24:10,592.592 INFO:tasks.rados.rados.0.mira097.stdout:554: expect (ObjNum 288 snap 0 seq_num 288) 2014-08-22 19:24:10,754.754 INFO:tasks.rados.rados.0.mira097.stdout:557: finishing write tid 7 to mira09725123-119 2014-08-22 19:24:10,755.755 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 119 v 765 (ObjNum 522 snap 8 seq_num 522) dirty exists 2014-08-22 19:24:10,755.755 INFO:tasks.rados.rados.0.mira097.stdout:571: finishing write tid 1 to mira09725123-85 2014-08-22 19:24:10,755.755 INFO:tasks.rados.rados.0.mira097.stdout:571: finishing write tid 2 to mira09725123-85 2014-08-22 19:24:10,755.755 INFO:tasks.rados.rados.0.mira097.stdout:571: finishing write tid 4 to mira09725123-85 2014-08-22 19:24:10,755.755 INFO:tasks.rados.rados.0.mira097.stdout:562: expect (ObjNum 251 snap 0 seq_num 251) 2014-08-22 19:24:10,770.770 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.1. 2014-08-22 19:24:10,773.773 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:03.683951 7f0bae71d700 20 client.4148.objecter shutdown clearing up homeless session... 2014-08-22 19:24:10,774.774 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:03.684304 7f0bae71d700 1 librados: shutdown 2014-08-22 19:24:10,778.778 INFO:tasks.thrashosds.thrasher:Added osd 1 2014-08-22 19:24:11,605.605 INFO:tasks.rados.rados.0.mira097.stdout:569: finishing write tid 4 to mira09725123-412 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 412 v 820 (ObjNum 523 snap 8 seq_num 523) dirty exists 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:566: finishing copy_from racing read to mira09725123-339 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:567: finishing copy_from racing read to mira09725123-52 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:553: done (15 left) 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:554: done (14 left) 2014-08-22 19:24:11,606.606 INFO:tasks.rados.rados.0.mira097.stdout:557: done (13 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:558: done (12 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:559: done (11 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:561: done (10 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:562: done (9 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:564: done (8 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:566: done (7 left) 2014-08-22 19:24:11,607.607 INFO:tasks.rados.rados.0.mira097.stdout:567: done (6 left) 2014-08-22 19:24:11,608.608 INFO:tasks.rados.rados.0.mira097.stdout:569: done (5 left) 2014-08-22 19:24:11,608.608 INFO:tasks.rados.rados.0.mira097.stdout:572: delete oid 95 current snap is 8 2014-08-22 19:24:11,817.817 INFO:tasks.rados.rados.0.mira097.stdout:565: finishing copy_from to mira09725123-100 2014-08-22 19:24:11,818.818 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 100 v 623 (ObjNum 458 snap 0 seq_num 458) dirty exists 2014-08-22 19:24:11,879.879 INFO:tasks.rados.rados.0.mira097.stdout:568: expect (ObjNum 363 snap 0 seq_num 363) 2014-08-22 19:24:12,293.293 INFO:tasks.rados.rados.0.mira097.stdout:565: finishing copy_from racing read to mira09725123-100 2014-08-22 19:24:12,293.293 INFO:tasks.rados.rados.0.mira097.stdout:560: finishing copy_from racing read to mira09725123-75 2014-08-22 19:24:12,293.293 INFO:tasks.rados.rados.0.mira097.stdout:571: finishing write tid 5 to mira09725123-85 2014-08-22 19:24:12,293.293 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 85 v 789 (ObjNum 525 snap 8 seq_num 525) dirty exists 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:570: finishing write tid 6 to mira09725123-170 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 170 v 782 (ObjNum 524 snap 8 seq_num 524) dirty exists 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:560: done (5 left) 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:565: done (4 left) 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:568: done (3 left) 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:570: done (2 left) 2014-08-22 19:24:12,294.294 INFO:tasks.rados.rados.0.mira097.stdout:571: done (1 left) 2014-08-22 19:24:12,295.295 INFO:tasks.rados.rados.0.mira097.stdout:572: done (0 left) 2014-08-22 19:24:12,295.295 INFO:tasks.rados.rados.0.mira097.stdout:573: read oid 422 snap -1 2014-08-22 19:24:12,295.295 INFO:tasks.rados.rados.0.mira097.stdout:574: snap_remove snap 3 2014-08-22 19:24:12,301.301 INFO:tasks.rados.rados.0.mira097.stdout:573: expect (ObjNum 421 snap 0 seq_num 421) 2014-08-22 19:24:15,011.011 INFO:tasks.rados.rados.0.mira097.stdout:573: done (1 left) 2014-08-22 19:24:15,011.011 INFO:tasks.rados.rados.0.mira097.stdout:574: done (0 left) 2014-08-22 19:24:15,012.012 INFO:tasks.rados.rados.0.mira097.stdout:575: read oid 69 snap -1 2014-08-22 19:24:15,012.012 INFO:tasks.rados.rados.0.mira097.stdout:576: read oid 164 snap -1 2014-08-22 19:24:15,014.014 INFO:tasks.rados.rados.0.mira097.stdout:577: write oid 446 current snap is 8 2014-08-22 19:24:15,015.015 INFO:tasks.rados.rados.0.mira097.stdout:577: seq_num 527 ranges {669275=400000} 2014-08-22 19:24:15,033.033 INFO:tasks.rados.rados.0.mira097.stdout:577: writing mira09725123-446 from 669275 to 1069275 tid 1 2014-08-22 19:24:15,033.033 INFO:tasks.rados.rados.0.mira097.stdout:578: delete oid 112 current snap is 8 2014-08-22 19:24:15,034.034 INFO:tasks.rados.rados.0.mira097.stdout:576: expect (ObjNum 163 snap 0 seq_num 163) 2014-08-22 19:24:15,461.461 INFO:tasks.rados.rados.0.mira097.stdout:577: finishing write tid 1 to mira09725123-446 2014-08-22 19:24:15,461.461 INFO:tasks.rados.rados.0.mira097.stdout:577: finishing write tid 3 to mira09725123-446 2014-08-22 19:24:15,468.468 INFO:tasks.rados.rados.0.mira097.stdout:577: finishing write tid 4 to mira09725123-446 2014-08-22 19:24:15,469.469 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 446 v 702 (ObjNum 527 snap 8 seq_num 527) dirty exists 2014-08-22 19:24:15,514.514 INFO:tasks.rados.rados.0.mira097.stdout:575: expect (ObjNum 68 snap 0 seq_num 68) 2014-08-22 19:24:15,778.778 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 0, 1] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:15,779.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:15,779.779 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:24:15,779.779 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:16,123.123 INFO:tasks.rados.rados.0.mira097.stdout:575: done (3 left) 2014-08-22 19:24:16,123.123 INFO:tasks.rados.rados.0.mira097.stdout:576: done (2 left) 2014-08-22 19:24:16,123.123 INFO:tasks.rados.rados.0.mira097.stdout:577: done (1 left) 2014-08-22 19:24:16,123.123 INFO:tasks.rados.rados.0.mira097.stdout:578: done (0 left) 2014-08-22 19:24:16,124.124 INFO:tasks.rados.rados.0.mira097.stdout:579: copy_from oid 8 from oid 163 current snap is 8 2014-08-22 19:24:16,124.124 INFO:tasks.rados.rados.0.mira097.stdout:580: snap_remove snap 6 2014-08-22 19:24:16,263.263 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.173758 7f1f85154700 1 librados: starting msgr at :/0 2014-08-22 19:24:16,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.173770 7f1f85154700 1 librados: starting objecter 2014-08-22 19:24:16,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.173990 7f1f85154700 1 librados: setting wanted keys 2014-08-22 19:24:16,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.173999 7f1f85154700 1 librados: calling monclient init 2014-08-22 19:24:16,279.279 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.189855 7f1f7e0ae700 10 client.?.objecter ms_handle_connect 0x7f1f800355e0 2014-08-22 19:24:16,279.279 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.189920 7f1f7e0ae700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:16,280.280 INFO:tasks.rados.rados.0.mira097.stdout:579: finishing copy_from to mira09725123-8 2014-08-22 19:24:16,280.280 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 8 v 208 (ObjNum 162 snap 0 seq_num 162) dirty exists 2014-08-22 19:24:16,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.201039 7f1f85154700 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:16,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.201260 7f1f85154700 1 librados: init done 2014-08-22 19:24:16,296.296 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.201988 7f1f7e0ae700 10 client.4159.objecter ms_dispatch 0x7f1f80000f80 osd_map(39..39 src has 1..39) v3 2014-08-22 19:24:16,296.296 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.201998 7f1f7e0ae700 3 client.4159.objecter handle_osd_map got epochs [39,39] > 0 2014-08-22 19:24:16,296.296 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.202002 7f1f7e0ae700 3 client.4159.objecter handle_osd_map decoding full epoch 39 2014-08-22 19:24:16,296.296 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.202138 7f1f7e0ae700 20 client.4159.objecter dump_active .. 0 homeless 2014-08-22 19:24:16,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.202189 7f1f7e0ae700 10 client.4159.objecter ms_dispatch 0x7f1f80000f80 osd_map(39..39 src has 1..39) v3 2014-08-22 19:24:16,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.202193 7f1f7e0ae700 3 client.4159.objecter handle_osd_map ignoring epochs [39,39] <= 39 2014-08-22 19:24:16,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.202196 7f1f7e0ae700 20 client.4159.objecter dump_active .. 0 homeless 2014-08-22 19:24:16,310.310 INFO:tasks.rados.rados.0.mira097.stdout:579: finishing copy_from racing read to mira09725123-8 2014-08-22 19:24:16,378.378 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:16,637.637 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.293139 7f1f85154700 20 client.4159.objecter shutdown clearing up homeless session... 2014-08-22 19:24:16,637.637 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:09.294111 7f1f85154700 1 librados: shutdown 2014-08-22 19:24:16,640.640 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:24:16,640.640 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 34' 2014-08-22 19:24:17,032.032 INFO:tasks.rados.rados.0.mira097.stdout:579: done (1 left) 2014-08-22 19:24:17,032.032 INFO:tasks.rados.rados.0.mira097.stdout:580: done (0 left) 2014-08-22 19:24:17,032.032 INFO:tasks.rados.rados.0.mira097.stdout:581: rollback oid 442 current snap is 8 2014-08-22 19:24:17,032.032 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 442 to 5 2014-08-22 19:24:17,032.032 INFO:tasks.rados.rados.0.mira097.stdout:582: write oid 391 current snap is 8 2014-08-22 19:24:17,033.033 INFO:tasks.rados.rados.0.mira097.stdout:582: seq_num 529 ranges {675287=495856,1834330=607859,3075286=1} 2014-08-22 19:24:17,059.059 INFO:tasks.rados.rados.0.mira097.stdout:582: writing mira09725123-391 from 675287 to 1171143 tid 1 2014-08-22 19:24:17,082.082 INFO:tasks.rados.rados.0.mira097.stdout:582: writing mira09725123-391 from 1834330 to 2442189 tid 2 2014-08-22 19:24:17,087.087 INFO:tasks.rados.rados.0.mira097.stdout:582: writing mira09725123-391 from 3075286 to 3075287 tid 3 2014-08-22 19:24:17,087.087 INFO:tasks.rados.rados.0.mira097.stdout:583: write oid 36 current snap is 8 2014-08-22 19:24:17,087.087 INFO:tasks.rados.rados.0.mira097.stdout:583: seq_num 530 ranges {770516=752333,2197038=526630,2770515=1} 2014-08-22 19:24:17,113.113 INFO:tasks.rados.rados.0.mira097.stdout:583: writing mira09725123-36 from 770516 to 1522849 tid 1 2014-08-22 19:24:17,133.133 INFO:tasks.rados.rados.0.mira097.stdout:583: writing mira09725123-36 from 2197038 to 2723668 tid 2 2014-08-22 19:24:17,134.134 INFO:tasks.rados.rados.0.mira097.stdout:583: writing mira09725123-36 from 2770515 to 2770516 tid 3 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:584: read oid 151 snap 0 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 442 v 724 (ObjNum 441 snap 0 seq_num 441) dirty exists 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:582: finishing write tid 1 to mira09725123-391 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:582: finishing write tid 2 to mira09725123-391 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:582: finishing write tid 3 to mira09725123-391 2014-08-22 19:24:17,137.137 INFO:tasks.rados.rados.0.mira097.stdout:582: finishing write tid 5 to mira09725123-391 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:582: finishing write tid 6 to mira09725123-391 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 391 v 850 (ObjNum 529 snap 8 seq_num 529) dirty exists 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:581: done (3 left) 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:582: done (2 left) 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:585: read oid 258 snap -1 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:586: read oid 199 snap -1 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:587: write oid 483 current snap is 8 2014-08-22 19:24:17,138.138 INFO:tasks.rados.rados.0.mira097.stdout:587: seq_num 531 ranges {715346=475561,1911170=748249,3414654=100692} 2014-08-22 19:24:17,153.153 INFO:tasks.rados.rados.0.mira097.stdout:587: writing mira09725123-483 from 715346 to 1190907 tid 1 2014-08-22 19:24:17,181.181 INFO:tasks.rados.rados.0.mira097.stdout:587: writing mira09725123-483 from 1911170 to 2659419 tid 2 2014-08-22 19:24:17,190.190 INFO:tasks.rados.rados.0.mira097.stdout:587: writing mira09725123-483 from 3414654 to 3515346 tid 3 2014-08-22 19:24:17,190.190 INFO:tasks.rados.rados.0.mira097.stdout:588: write oid 45 current snap is 8 2014-08-22 19:24:17,190.190 INFO:tasks.rados.rados.0.mira097.stdout:588: seq_num 532 ranges {774479=1} 2014-08-22 19:24:17,195.195 INFO:tasks.rados.rados.0.mira097.stdout:588: writing mira09725123-45 from 774479 to 774480 tid 1 2014-08-22 19:24:17,195.195 INFO:tasks.rados.rados.0.mira097.stdout:589: write oid 16 current snap is 8 2014-08-22 19:24:17,195.195 INFO:tasks.rados.rados.0.mira097.stdout:589: seq_num 533 ranges {470733=1} 2014-08-22 19:24:17,198.198 INFO:tasks.rados.rados.0.mira097.stdout:589: writing mira09725123-16 from 470733 to 470734 tid 1 2014-08-22 19:24:17,198.198 INFO:tasks.rados.rados.0.mira097.stdout:590: rollback oid 210 current snap is 8 2014-08-22 19:24:17,199.199 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 210 to 5 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:583: finishing write tid 1 to mira09725123-36 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:583: finishing write tid 2 to mira09725123-36 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:583: finishing write tid 3 to mira09725123-36 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:583: finishing write tid 5 to mira09725123-36 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:591: rollback oid 337 current snap is 8 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 337 to 7 2014-08-22 19:24:17,200.200 INFO:tasks.rados.rados.0.mira097.stdout:592: snap_remove snap 1 2014-08-22 19:24:17,206.206 INFO:tasks.rados.rados.0.mira097.stdout:583: finishing write tid 6 to mira09725123-36 2014-08-22 19:24:17,206.206 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 36 v 860 (ObjNum 530 snap 8 seq_num 530) dirty exists 2014-08-22 19:24:17,241.241 INFO:tasks.rados.rados.0.mira097.stdout:584: expect (ObjNum 150 snap 0 seq_num 150) 2014-08-22 19:24:17,299.299 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.208933 7f6e75e8f700 1 librados: starting msgr at :/0 2014-08-22 19:24:17,299.299 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.208940 7f6e75e8f700 1 librados: starting objecter 2014-08-22 19:24:17,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.209080 7f6e75e8f700 1 librados: setting wanted keys 2014-08-22 19:24:17,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.209086 7f6e75e8f700 1 librados: calling monclient init 2014-08-22 19:24:17,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.209867 7f6e6e7fc700 10 client.?.objecter ms_handle_connect 0x7f6e700355e0 2014-08-22 19:24:17,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.209911 7f6e6e7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:17,301.301 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211400 7f6e75e8f700 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:17,301.301 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211491 7f6e75e8f700 1 librados: init done 2014-08-22 19:24:17,301.301 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211768 7f6e6e7fc700 10 client.4149.objecter ms_dispatch 0x7f6e70000f80 osd_map(40..40 src has 1..40) v3 2014-08-22 19:24:17,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211777 7f6e6e7fc700 3 client.4149.objecter handle_osd_map got epochs [40,40] > 0 2014-08-22 19:24:17,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211780 7f6e6e7fc700 3 client.4149.objecter handle_osd_map decoding full epoch 40 2014-08-22 19:24:17,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211878 7f6e6e7fc700 20 client.4149.objecter dump_active .. 0 homeless 2014-08-22 19:24:17,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211922 7f6e6e7fc700 10 client.4149.objecter ms_dispatch 0x7f6e70000f80 osd_map(40..40 src has 1..40) v3 2014-08-22 19:24:17,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211928 7f6e6e7fc700 3 client.4149.objecter handle_osd_map ignoring epochs [40,40] <= 40 2014-08-22 19:24:17,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:10.211931 7f6e6e7fc700 20 client.4149.objecter dump_active .. 0 homeless 2014-08-22 19:24:18,014.014 INFO:tasks.rados.rados.0.mira097.stdout:585: expect (ObjNum 257 snap 0 seq_num 257) 2014-08-22 19:24:18,091.091 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 34 2014-08-22 19:24:18,094.094 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:11.005048 7f6e75e8f700 20 client.4149.objecter shutdown clearing up homeless session... 2014-08-22 19:24:18,094.094 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:11.005340 7f6e75e8f700 1 librados: shutdown 2014-08-22 19:24:18,310.310 INFO:tasks.rados.rados.0.mira097.stdout:586: expect (ObjNum 198 snap 0 seq_num 198) 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 337 v 865 (ObjNum 336 snap 0 seq_num 336) dirty exists 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 210 v 868 (ObjNum 209 snap 0 seq_num 209) dirty exists 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:589: finishing write tid 1 to mira09725123-16 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:589: finishing write tid 3 to mira09725123-16 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:589: finishing write tid 4 to mira09725123-16 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 16 v 854 (ObjNum 533 snap 8 seq_num 533) dirty exists 2014-08-22 19:24:18,321.321 INFO:tasks.rados.rados.0.mira097.stdout:583: done (9 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:584: done (8 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:585: done (7 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:586: done (6 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:589: done (5 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:590: done (4 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:591: done (3 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:592: done (2 left) 2014-08-22 19:24:18,322.322 INFO:tasks.rados.rados.0.mira097.stdout:593: write oid 165 current snap is 8 2014-08-22 19:24:18,323.323 INFO:tasks.rados.rados.0.mira097.stdout:593: seq_num 534 ranges {415563=428094,1215562=1} 2014-08-22 19:24:18,334.334 INFO:tasks.rados.rados.0.mira097.stdout:593: writing mira09725123-165 from 415563 to 843657 tid 1 2014-08-22 19:24:18,338.338 INFO:tasks.rados.rados.0.mira097.stdout:593: writing mira09725123-165 from 1215562 to 1215563 tid 2 2014-08-22 19:24:18,339.339 INFO:tasks.rados.rados.0.mira097.stdout:594: delete oid 289 current snap is 8 2014-08-22 19:24:18,410.410 INFO:tasks.rados.rados.0.mira097.stdout:594: done (3 left) 2014-08-22 19:24:18,410.410 INFO:tasks.rados.rados.0.mira097.stdout:595: rollback oid 341 current snap is 8 2014-08-22 19:24:18,410.410 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 341 to 7 2014-08-22 19:24:18,410.410 INFO:tasks.rados.rados.0.mira097.stdout:596: snap_create 2014-08-22 19:24:18,410.410 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:24:18,424.424 INFO:tasks.rados.rados.0.mira097.stdout:593: finishing write tid 1 to mira09725123-165 2014-08-22 19:24:18,424.424 INFO:tasks.rados.rados.0.mira097.stdout:593: finishing write tid 2 to mira09725123-165 2014-08-22 19:24:18,424.424 INFO:tasks.rados.rados.0.mira097.stdout:593: finishing write tid 4 to mira09725123-165 2014-08-22 19:24:18,628.628 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 341 v 887 (ObjNum 340 snap 0 seq_num 340) dirty exists 2014-08-22 19:24:18,628.628 INFO:tasks.rados.rados.0.mira097.stdout:595: done (3 left) 2014-08-22 19:24:18,628.628 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:24:18,925.925 INFO:tasks.rados.rados.0.mira097.stdout:593: finishing write tid 5 to mira09725123-165 2014-08-22 19:24:18,925.925 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 165 v 885 (ObjNum 534 snap 8 seq_num 534) dirty exists 2014-08-22 19:24:18,925.925 INFO:tasks.rados.rados.0.mira097.stdout:593: done (2 left) 2014-08-22 19:24:18,926.926 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:21,053.053 INFO:tasks.rados.rados.0.mira097.stdout:588: finishing write tid 1 to mira09725123-45 2014-08-22 19:24:21,054.054 INFO:tasks.rados.rados.0.mira097.stdout:588: finishing write tid 3 to mira09725123-45 2014-08-22 19:24:21,054.054 INFO:tasks.rados.rados.0.mira097.stdout:588: finishing write tid 4 to mira09725123-45 2014-08-22 19:24:21,055.055 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 45 v 801 (ObjNum 532 snap 8 seq_num 532) dirty exists 2014-08-22 19:24:21,055.055 INFO:tasks.rados.rados.0.mira097.stdout:588: done (1 left) 2014-08-22 19:24:21,055.055 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:21,399.399 INFO:tasks.rados.rados.0.mira097.stdout:587: finishing write tid 1 to mira09725123-483 2014-08-22 19:24:21,399.399 INFO:tasks.rados.rados.0.mira097.stdout:587: finishing write tid 2 to mira09725123-483 2014-08-22 19:24:21,399.399 INFO:tasks.rados.rados.0.mira097.stdout:587: finishing write tid 3 to mira09725123-483 2014-08-22 19:24:21,400.400 INFO:tasks.rados.rados.0.mira097.stdout:587: finishing write tid 5 to mira09725123-483 2014-08-22 19:24:21,425.425 INFO:tasks.rados.rados.0.mira097.stdout:587: finishing write tid 6 to mira09725123-483 2014-08-22 19:24:21,425.425 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 483 v 878 (ObjNum 531 snap 8 seq_num 531) dirty exists 2014-08-22 19:24:21,425.425 INFO:tasks.rados.rados.0.mira097.stdout:587: done (0 left) 2014-08-22 19:24:21,819.819 INFO:tasks.rados.rados.0.mira097.stdout:596: done (0 left) 2014-08-22 19:24:21,819.819 INFO:tasks.rados.rados.0.mira097.stdout:597: read oid 108 snap -1 2014-08-22 19:24:21,819.819 INFO:tasks.rados.rados.0.mira097.stdout:598: write oid 347 current snap is 9 2014-08-22 19:24:21,819.819 INFO:tasks.rados.rados.0.mira097.stdout:598: seq_num 536 ranges {455623=771703,1655622=1} 2014-08-22 19:24:21,839.839 INFO:tasks.rados.rados.0.mira097.stdout:598: writing mira09725123-347 from 455623 to 1227326 tid 1 2014-08-22 19:24:21,842.842 INFO:tasks.rados.rados.0.mira097.stdout:598: writing mira09725123-347 from 1655622 to 1655623 tid 2 2014-08-22 19:24:21,842.842 INFO:tasks.rados.rados.0.mira097.stdout:599: snap_remove snap 0 2014-08-22 19:24:21,905.905 INFO:tasks.rados.rados.0.mira097.stdout:598: finishing write tid 1 to mira09725123-347 2014-08-22 19:24:21,906.906 INFO:tasks.rados.rados.0.mira097.stdout:598: finishing write tid 2 to mira09725123-347 2014-08-22 19:24:21,906.906 INFO:tasks.rados.rados.0.mira097.stdout:598: finishing write tid 4 to mira09725123-347 2014-08-22 19:24:21,925.925 INFO:tasks.rados.rados.0.mira097.stdout:598: finishing write tid 5 to mira09725123-347 2014-08-22 19:24:21,925.925 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 347 v 887 (ObjNum 536 snap 9 seq_num 536) dirty exists 2014-08-22 19:24:22,988.988 INFO:tasks.rados.rados.0.mira097.stdout:597: expect (ObjNum 107 snap 0 seq_num 107) 2014-08-22 19:24:23,098.098 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 0, 1] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:23,098.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:23,098.098 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-22 19:24:23,099.099 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:24:23,099.099 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:24:23,099.099 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-22 19:24:24,682.682 INFO:tasks.rados.rados.0.mira097.stdout:597: done (2 left) 2014-08-22 19:24:24,683.683 INFO:tasks.rados.rados.0.mira097.stdout:598: done (1 left) 2014-08-22 19:24:24,683.683 INFO:tasks.rados.rados.0.mira097.stdout:599: done (0 left) 2014-08-22 19:24:24,683.683 INFO:tasks.rados.rados.0.mira097.stdout:600: snap_remove snap 5 2014-08-22 19:24:24,924.924 INFO:tasks.rados.rados.0.mira097.stdout:600: done (0 left) 2014-08-22 19:24:24,924.924 INFO:tasks.rados.rados.0.mira097.stdout:601: delete oid 270 current snap is 9 2014-08-22 19:24:24,977.977 INFO:tasks.rados.rados.0.mira097.stdout:601: done (0 left) 2014-08-22 19:24:24,978.978 INFO:tasks.rados.rados.0.mira097.stdout:602: snap_remove snap 7 2014-08-22 19:24:25,945.945 INFO:tasks.rados.rados.0.mira097.stdout:602: done (0 left) 2014-08-22 19:24:25,945.945 INFO:tasks.rados.rados.0.mira097.stdout:603: read oid 77 snap -1 2014-08-22 19:24:25,945.945 INFO:tasks.rados.rados.0.mira097.stdout:604: snap_create 2014-08-22 19:24:25,945.945 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:25,957.957 INFO:tasks.rados.rados.0.mira097.stdout:603: expect (ObjNum 515 snap 5 seq_num 515) 2014-08-22 19:24:27,189.189 INFO:tasks.rados.rados.0.mira097.stdout:603: done (0 left) 2014-08-22 19:24:27,466.466 INFO:tasks.rados.rados.0.mira097.stdout:604: done (0 left) 2014-08-22 19:24:27,466.466 INFO:tasks.rados.rados.0.mira097.stdout:605: read oid 448 snap -1 2014-08-22 19:24:27,466.466 INFO:tasks.rados.rados.0.mira097.stdout:606: read oid 164 snap -1 2014-08-22 19:24:27,471.471 INFO:tasks.rados.rados.0.mira097.stdout:607: copy_from oid 27 from oid 116 current snap is 10 2014-08-22 19:24:27,471.471 INFO:tasks.rados.rados.0.mira097.stdout:608: read oid 279 snap -1 2014-08-22 19:24:27,471.471 INFO:tasks.rados.rados.0.mira097.stdout:609: rollback oid 326 current snap is 10 2014-08-22 19:24:27,471.471 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 326 to 9 2014-08-22 19:24:27,471.471 INFO:tasks.rados.rados.0.mira097.stdout:610: write oid 16 current snap is 10 2014-08-22 19:24:27,472.472 INFO:tasks.rados.rados.0.mira097.stdout:610: seq_num 538 ranges {460610=467759,1616335=625088,2982464=411952,3660609=1} 2014-08-22 19:24:27,488.488 INFO:tasks.rados.rados.0.mira097.stdout:610: writing mira09725123-16 from 460610 to 928369 tid 1 2014-08-22 19:24:27,513.513 INFO:tasks.rados.rados.0.mira097.stdout:610: writing mira09725123-16 from 1616335 to 2241423 tid 2 2014-08-22 19:24:27,532.532 INFO:tasks.rados.rados.0.mira097.stdout:610: writing mira09725123-16 from 2982464 to 3394416 tid 3 2014-08-22 19:24:27,535.535 INFO:tasks.rados.rados.0.mira097.stdout:610: writing mira09725123-16 from 3660609 to 3660610 tid 4 2014-08-22 19:24:27,537.537 INFO:tasks.rados.rados.0.mira097.stdout:611: delete oid 286 current snap is 10 2014-08-22 19:24:27,537.537 INFO:tasks.rados.rados.0.mira097.stdout:606: expect (ObjNum 163 snap 0 seq_num 163) 2014-08-22 19:24:27,546.546 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 326 v 934 (ObjNum 325 snap 0 seq_num 325) dirty exists 2014-08-22 19:24:27,638.638 INFO:tasks.rados.rados.0.mira097.stdout:607: finishing copy_from to mira09725123-27 2014-08-22 19:24:27,639.639 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 27 v 141 (ObjNum 115 snap 0 seq_num 115) dirty exists 2014-08-22 19:24:27,639.639 INFO:tasks.rados.rados.0.mira097.stdout:607: finishing copy_from racing read to mira09725123-27 2014-08-22 19:24:27,639.639 INFO:tasks.rados.rados.0.mira097.stdout:605: expect (ObjNum 447 snap 0 seq_num 447) 2014-08-22 19:24:28,518.518 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 1 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 2 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 3 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 4 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 6 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:610: finishing write tid 7 to mira09725123-16 2014-08-22 19:24:28,519.519 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 16 v 930 (ObjNum 538 snap 10 seq_num 538) dirty exists 2014-08-22 19:24:28,520.520 INFO:tasks.rados.rados.0.mira097.stdout:608: expect (ObjNum 278 snap 0 seq_num 278) 2014-08-22 19:24:28,694.694 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 0, 1] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:28,694.694 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:24:29,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.051462 7fecf7bb2700 1 librados: starting msgr at :/0 2014-08-22 19:24:29,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.051471 7fecf7bb2700 1 librados: starting objecter 2014-08-22 19:24:29,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.051624 7fecf7bb2700 1 librados: setting wanted keys 2014-08-22 19:24:29,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.051628 7fecf7bb2700 1 librados: calling monclient init 2014-08-22 19:24:29,143.143 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.053388 7fecf4c5f700 10 client.?.objecter ms_handle_connect 0x7fecf00355e0 2014-08-22 19:24:29,143.143 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.053427 7fecf4c5f700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:29,146.146 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.056600 7fecf7bb2700 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:29,146.146 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.056665 7fecf7bb2700 1 librados: init done 2014-08-22 19:24:29,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057643 7fecf4c5f700 10 client.4162.objecter ms_dispatch 0x7fecf0000f80 osd_map(47..47 src has 1..47) v3 2014-08-22 19:24:29,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057651 7fecf4c5f700 3 client.4162.objecter handle_osd_map got epochs [47,47] > 0 2014-08-22 19:24:29,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057653 7fecf4c5f700 3 client.4162.objecter handle_osd_map decoding full epoch 47 2014-08-22 19:24:29,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057722 7fecf4c5f700 20 client.4162.objecter dump_active .. 0 homeless 2014-08-22 19:24:29,148.148 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057751 7fecf4c5f700 10 client.4162.objecter ms_dispatch 0x7fecf0000f80 osd_map(47..47 src has 1..47) v3 2014-08-22 19:24:29,148.148 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057753 7fecf4c5f700 3 client.4162.objecter handle_osd_map ignoring epochs [47,47] <= 47 2014-08-22 19:24:29,148.148 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.057755 7fecf4c5f700 20 client.4162.objecter dump_active .. 0 homeless 2014-08-22 19:24:29,488.488 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:24:29,492.492 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.401862 7fecf7bb2700 20 client.4162.objecter shutdown clearing up homeless session... 2014-08-22 19:24:29,492.492 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:22.402158 7fecf7bb2700 1 librados: shutdown 2014-08-22 19:24:29,496.496 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021062 7fef694dd700 1 librados: starting msgr at :/0 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021095 7fef694dd700 1 librados: starting objecter 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021235 7fef694dd700 1 librados: setting wanted keys 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021240 7fef694dd700 1 librados: calling monclient init 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021934 7fef62559700 10 client.?.objecter ms_handle_connect 0x7fef640355e0 2014-08-22 19:24:30,111.111 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.021955 7fef62559700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:30,112.112 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023081 7fef694dd700 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:30,112.112 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023175 7fef694dd700 1 librados: init done 2014-08-22 19:24:30,112.112 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023549 7fef62559700 10 client.4152.objecter ms_dispatch 0x7fef64000f80 osd_map(48..48 src has 1..48) v3 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023556 7fef62559700 3 client.4152.objecter handle_osd_map got epochs [48,48] > 0 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023557 7fef62559700 3 client.4152.objecter handle_osd_map decoding full epoch 48 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023623 7fef62559700 20 client.4152.objecter dump_active .. 0 homeless 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023651 7fef62559700 10 client.4152.objecter ms_dispatch 0x7fef64000f80 osd_map(48..48 src has 1..48) v3 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023653 7fef62559700 3 client.4152.objecter handle_osd_map ignoring epochs [48,48] <= 48 2014-08-22 19:24:30,113.113 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.023654 7fef62559700 20 client.4152.objecter dump_active .. 0 homeless 2014-08-22 19:24:30,505.505 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:24:30,510.510 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.419521 7fef694dd700 20 client.4152.objecter shutdown clearing up homeless session... 2014-08-22 19:24:30,510.510 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:23.419995 7fef694dd700 1 librados: shutdown 2014-08-22 19:24:30,515.515 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:24:31,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.056913 7eff6228f700 1 librados: starting msgr at :/0 2014-08-22 19:24:31,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.056933 7eff6228f700 1 librados: starting objecter 2014-08-22 19:24:31,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.057175 7eff6228f700 1 librados: setting wanted keys 2014-08-22 19:24:31,148.148 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.057189 7eff6228f700 1 librados: calling monclient init 2014-08-22 19:24:31,149.149 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.059613 7eff5affd700 10 client.?.objecter ms_handle_connect 0x7eff5c0355e0 2014-08-22 19:24:31,149.149 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.059848 7eff5affd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:31,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.063059 7eff6228f700 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:31,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.063283 7eff6228f700 1 librados: init done 2014-08-22 19:24:31,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064073 7eff5affd700 10 client.4165.objecter ms_dispatch 0x7eff5c000f80 osd_map(49..49 src has 1..49) v3 2014-08-22 19:24:31,154.154 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064089 7eff5affd700 3 client.4165.objecter handle_osd_map got epochs [49,49] > 0 2014-08-22 19:24:31,154.154 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064095 7eff5affd700 3 client.4165.objecter handle_osd_map decoding full epoch 49 2014-08-22 19:24:31,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064260 7eff5affd700 20 client.4165.objecter dump_active .. 0 homeless 2014-08-22 19:24:31,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064350 7eff5affd700 10 client.4165.objecter ms_dispatch 0x7eff5c000f80 osd_map(49..49 src has 1..49) v3 2014-08-22 19:24:31,232.232 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064362 7eff5affd700 3 client.4165.objecter handle_osd_map ignoring epochs [49,49] <= 49 2014-08-22 19:24:31,232.232 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.064369 7eff5affd700 20 client.4165.objecter dump_active .. 0 homeless 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:605: done (6 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:606: done (5 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:607: done (4 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:608: done (3 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:609: done (2 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:610: done (1 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:611: done (0 left) 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:612: write oid 151 current snap is 10 2014-08-22 19:24:31,247.247 INFO:tasks.rados.rados.0.mira097.stdout:612: seq_num 540 ranges {99645=1} 2014-08-22 19:24:31,248.248 INFO:tasks.rados.rados.0.mira097.stdout:612: writing mira09725123-151 from 99645 to 99646 tid 1 2014-08-22 19:24:31,248.248 INFO:tasks.rados.rados.0.mira097.stdout:613: read oid 14 snap -1 2014-08-22 19:24:31,248.248 INFO:tasks.rados.rados.0.mira097.stdout:614: read oid 343 snap 8 2014-08-22 19:24:31,248.248 INFO:tasks.rados.rados.0.mira097.stdout:615: delete oid 235 current snap is 10 2014-08-22 19:24:31,350.350 INFO:tasks.rados.rados.0.mira097.stdout:614: expect (ObjNum 342 snap 0 seq_num 342) 2014-08-22 19:24:31,561.561 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:24:31,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.474873 7eff6228f700 20 client.4165.objecter shutdown clearing up homeless session... 2014-08-22 19:24:31,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:24.475305 7eff6228f700 1 librados: shutdown 2014-08-22 19:24:31,569.569 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:24:31,966.966 INFO:tasks.rados.rados.0.mira097.stdout:612: finishing write tid 1 to mira09725123-151 2014-08-22 19:24:31,966.966 INFO:tasks.rados.rados.0.mira097.stdout:612: finishing write tid 3 to mira09725123-151 2014-08-22 19:24:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:612: finishing write tid 4 to mira09725123-151 2014-08-22 19:24:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 151 v 937 (ObjNum 540 snap 10 seq_num 540) dirty exists 2014-08-22 19:24:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:613: expect (ObjNum 13 snap 0 seq_num 13) 2014-08-22 19:24:32,218.218 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.128362 7f5669e48700 1 librados: starting msgr at :/0 2014-08-22 19:24:32,218.218 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.128367 7f5669e48700 1 librados: starting objecter 2014-08-22 19:24:32,218.218 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.128506 7f5669e48700 1 librados: setting wanted keys 2014-08-22 19:24:32,219.219 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.128510 7f5669e48700 1 librados: calling monclient init 2014-08-22 19:24:32,219.219 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.130355 7f56627fc700 10 client.?.objecter ms_handle_connect 0x7f56640355e0 2014-08-22 19:24:32,220.220 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.130374 7f56627fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:32,222.222 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.133368 7f5669e48700 10 client.4168.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:32,222.222 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.133449 7f5669e48700 1 librados: init done 2014-08-22 19:24:32,223.223 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134347 7f56627fc700 10 client.4168.objecter ms_dispatch 0x7f5664000f80 osd_map(50..50 src has 1..50) v3 2014-08-22 19:24:32,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134360 7f56627fc700 3 client.4168.objecter handle_osd_map got epochs [50,50] > 0 2014-08-22 19:24:32,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134364 7f56627fc700 3 client.4168.objecter handle_osd_map decoding full epoch 50 2014-08-22 19:24:32,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134432 7f56627fc700 20 client.4168.objecter dump_active .. 0 homeless 2014-08-22 19:24:32,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134464 7f56627fc700 10 client.4168.objecter ms_dispatch 0x7f5664000f80 osd_map(50..50 src has 1..50) v3 2014-08-22 19:24:32,225.225 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134466 7f56627fc700 3 client.4168.objecter handle_osd_map ignoring epochs [50,50] <= 50 2014-08-22 19:24:32,225.225 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.134468 7f56627fc700 20 client.4168.objecter dump_active .. 0 homeless 2014-08-22 19:24:32,566.566 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:24:32,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.480372 7f5669e48700 20 client.4168.objecter shutdown clearing up homeless session... 2014-08-22 19:24:32,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:25.480746 7f5669e48700 1 librados: shutdown 2014-08-22 19:24:32,574.574 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:24:32,947.947 INFO:tasks.rados.rados.0.mira097.stdout:612: done (3 left) 2014-08-22 19:24:32,947.947 INFO:tasks.rados.rados.0.mira097.stdout:613: done (2 left) 2014-08-22 19:24:32,948.948 INFO:tasks.rados.rados.0.mira097.stdout:614: done (1 left) 2014-08-22 19:24:32,948.948 INFO:tasks.rados.rados.0.mira097.stdout:615: done (0 left) 2014-08-22 19:24:32,948.948 INFO:tasks.rados.rados.0.mira097.stdout:616: delete oid 246 current snap is 10 2014-08-22 19:24:32,961.961 INFO:tasks.rados.rados.0.mira097.stdout:616: done (0 left) 2014-08-22 19:24:32,961.961 INFO:tasks.rados.rados.0.mira097.stdout:617: delete oid 288 current snap is 10 2014-08-22 19:24:32,973.973 INFO:tasks.rados.rados.0.mira097.stdout:617: done (0 left) 2014-08-22 19:24:32,973.973 INFO:tasks.rados.rados.0.mira097.stdout:618: snap_remove snap 8 2014-08-22 19:24:33,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.123799 7f6d75559700 1 librados: starting msgr at :/0 2014-08-22 19:24:33,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.123811 7f6d75559700 1 librados: starting objecter 2014-08-22 19:24:33,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.124073 7f6d75559700 1 librados: setting wanted keys 2014-08-22 19:24:33,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.124087 7f6d75559700 1 librados: calling monclient init 2014-08-22 19:24:33,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.126011 7f6d6e58b700 10 client.?.objecter ms_handle_connect 0x7f6d700355e0 2014-08-22 19:24:33,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.126064 7f6d6e58b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:33,218.218 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129065 7f6d75559700 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:33,218.218 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129219 7f6d75559700 1 librados: init done 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129679 7f6d6e58b700 10 client.4155.objecter ms_dispatch 0x7f6d70000f80 osd_map(51..51 src has 1..51) v3 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129690 7f6d6e58b700 3 client.4155.objecter handle_osd_map got epochs [51,51] > 0 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129693 7f6d6e58b700 3 client.4155.objecter handle_osd_map decoding full epoch 51 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.129871 7f6d6e58b700 20 client.4155.objecter dump_active .. 0 homeless 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.130222 7f6d6e58b700 10 client.4155.objecter ms_dispatch 0x7f6d70000f80 osd_map(51..51 src has 1..51) v3 2014-08-22 19:24:33,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.130229 7f6d6e58b700 3 client.4155.objecter handle_osd_map ignoring epochs [51,51] <= 51 2014-08-22 19:24:33,254.254 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.130233 7f6d6e58b700 20 client.4155.objecter dump_active .. 0 homeless 2014-08-22 19:24:33,548.548 INFO:tasks.rados.rados.0.mira097.stdout:618: done (0 left) 2014-08-22 19:24:33,548.548 INFO:tasks.rados.rados.0.mira097.stdout:619: snap_create 2014-08-22 19:24:33,572.572 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:24:33,577.577 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.486351 7f6d75559700 20 client.4155.objecter shutdown clearing up homeless session... 2014-08-22 19:24:33,577.577 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:26.487151 7f6d75559700 1 librados: shutdown 2014-08-22 19:24:33,585.585 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:24:34,226.226 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.136036 7f6239c8c700 1 librados: starting msgr at :/0 2014-08-22 19:24:34,227.227 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.136055 7f6239c8c700 1 librados: starting objecter 2014-08-22 19:24:34,227.227 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.136319 7f6239c8c700 1 librados: setting wanted keys 2014-08-22 19:24:34,227.227 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.136333 7f6239c8c700 1 librados: calling monclient init 2014-08-22 19:24:34,228.228 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.138631 7f62327fc700 10 client.?.objecter ms_handle_connect 0x7f62340355e0 2014-08-22 19:24:34,228.228 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.138673 7f62327fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:34,233.233 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.142429 7f6239c8c700 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:34,233.233 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.142622 7f6239c8c700 1 librados: init done 2014-08-22 19:24:34,234.234 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143507 7f62327fc700 10 client.4171.objecter ms_dispatch 0x7f6234000f80 osd_map(52..52 src has 1..52) v3 2014-08-22 19:24:34,234.234 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143517 7f62327fc700 3 client.4171.objecter handle_osd_map got epochs [52,52] > 0 2014-08-22 19:24:34,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143520 7f62327fc700 3 client.4171.objecter handle_osd_map decoding full epoch 52 2014-08-22 19:24:34,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143645 7f62327fc700 20 client.4171.objecter dump_active .. 0 homeless 2014-08-22 19:24:34,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143696 7f62327fc700 10 client.4171.objecter ms_dispatch 0x7f6234000f80 osd_map(52..52 src has 1..52) v3 2014-08-22 19:24:34,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143700 7f62327fc700 3 client.4171.objecter handle_osd_map ignoring epochs [52,52] <= 52 2014-08-22 19:24:34,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.143703 7f62327fc700 20 client.4171.objecter dump_active .. 0 homeless 2014-08-22 19:24:34,562.562 INFO:tasks.rados.rados.0.mira097.stdout:619: done (0 left) 2014-08-22 19:24:34,562.562 INFO:tasks.rados.rados.0.mira097.stdout:620: snap_create 2014-08-22 19:24:34,587.587 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:24:34,593.593 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.503672 7f6239c8c700 20 client.4171.objecter shutdown clearing up homeless session... 2014-08-22 19:24:34,594.594 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:27.504466 7f6239c8c700 1 librados: shutdown 2014-08-22 19:24:34,602.602 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:24:34,602.602 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:35,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.149317 7fddd0831700 1 librados: starting msgr at :/0 2014-08-22 19:24:35,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.149325 7fddd0831700 1 librados: starting objecter 2014-08-22 19:24:35,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.149621 7fddd0831700 1 librados: setting wanted keys 2014-08-22 19:24:35,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.149630 7fddd0831700 1 librados: calling monclient init 2014-08-22 19:24:35,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.151149 7fddc98ad700 10 client.?.objecter ms_handle_connect 0x7fddcc0355e0 2014-08-22 19:24:35,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.151273 7fddc98ad700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:35,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.154448 7fddd0831700 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:35,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.154649 7fddd0831700 1 librados: init done 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155122 7fddc98ad700 10 client.4151.objecter ms_dispatch 0x7fddcc000f80 osd_map(53..53 src has 1..53) v3 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155142 7fddc98ad700 3 client.4151.objecter handle_osd_map got epochs [53,53] > 0 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155147 7fddc98ad700 3 client.4151.objecter handle_osd_map decoding full epoch 53 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155315 7fddc98ad700 20 client.4151.objecter dump_active .. 0 homeless 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155403 7fddc98ad700 10 client.4151.objecter ms_dispatch 0x7fddcc000f80 osd_map(53..53 src has 1..53) v3 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155417 7fddc98ad700 3 client.4151.objecter handle_osd_map ignoring epochs [53,53] <= 53 2014-08-22 19:24:35,245.245 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.155423 7fddc98ad700 20 client.4151.objecter dump_active .. 0 homeless 2014-08-22 19:24:35,419.419 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:35,607.607 INFO:tasks.rados.rados.0.mira097.stdout:620: done (0 left) 2014-08-22 19:24:35,607.607 INFO:tasks.rados.rados.0.mira097.stdout:621: read oid 382 snap -1 2014-08-22 19:24:35,607.607 INFO:tasks.rados.rados.0.mira097.stdout:622: delete oid 170 current snap is 12 2014-08-22 19:24:35,607.607 INFO:tasks.rados.rados.0.mira097.stdout:622: done (1 left) 2014-08-22 19:24:35,607.607 INFO:tasks.rados.rados.0.mira097.stdout:623: snap_remove snap 11 2014-08-22 19:24:35,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.256010 7fddd0831700 20 client.4151.objecter shutdown clearing up homeless session... 2014-08-22 19:24:35,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:28.256842 7fddd0831700 1 librados: shutdown 2014-08-22 19:24:35,611.611 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:35,611.611 INFO:tasks.rados.rados.0.mira097.stdout:621: expect (ObjNum 381 snap 0 seq_num 381) 2014-08-22 19:24:36,228.228 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.137538 7ff86abc8700 1 librados: starting msgr at :/0 2014-08-22 19:24:36,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.137545 7ff86abc8700 1 librados: starting objecter 2014-08-22 19:24:36,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.137703 7ff86abc8700 1 librados: setting wanted keys 2014-08-22 19:24:36,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.137710 7ff86abc8700 1 librados: calling monclient init 2014-08-22 19:24:36,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.138560 7ff85b7fe700 10 client.?.objecter ms_handle_connect 0x7ff8640355e0 2014-08-22 19:24:36,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.138580 7ff85b7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.139922 7ff86abc8700 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140035 7ff86abc8700 1 librados: init done 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140281 7ff85b7fe700 10 client.4154.objecter ms_dispatch 0x7ff864000f80 osd_map(54..54 src has 1..54) v3 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140289 7ff85b7fe700 3 client.4154.objecter handle_osd_map got epochs [54,54] > 0 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140292 7ff85b7fe700 3 client.4154.objecter handle_osd_map decoding full epoch 54 2014-08-22 19:24:36,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140376 7ff85b7fe700 20 client.4154.objecter dump_active .. 0 homeless 2014-08-22 19:24:36,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140522 7ff85b7fe700 10 client.4154.objecter ms_dispatch 0x7ff864000f80 osd_map(54..54 src has 1..54) v3 2014-08-22 19:24:36,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140526 7ff85b7fe700 3 client.4154.objecter handle_osd_map ignoring epochs [54,54] <= 54 2014-08-22 19:24:36,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.140527 7ff85b7fe700 20 client.4154.objecter dump_active .. 0 homeless 2014-08-22 19:24:36,314.314 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:36,665.665 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.228921 7ff86abc8700 20 client.4154.objecter shutdown clearing up homeless session... 2014-08-22 19:24:36,665.665 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:29.229269 7ff86abc8700 1 librados: shutdown 2014-08-22 19:24:36,665.665 INFO:tasks.rados.rados.0.mira097.stdout:621: done (1 left) 2014-08-22 19:24:36,666.666 INFO:tasks.rados.rados.0.mira097.stdout:623: done (0 left) 2014-08-22 19:24:36,666.666 INFO:tasks.rados.rados.0.mira097.stdout:624: snap_remove snap 10 2014-08-22 19:24:36,669.669 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:24:37,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.217342 7fbf49d8e700 1 librados: starting msgr at :/0 2014-08-22 19:24:37,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.217355 7fbf49d8e700 1 librados: starting objecter 2014-08-22 19:24:37,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.217642 7fbf49d8e700 1 librados: setting wanted keys 2014-08-22 19:24:37,307.307 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.217652 7fbf49d8e700 1 librados: calling monclient init 2014-08-22 19:24:37,309.309 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.219381 7fbf427fc700 10 client.?.objecter ms_handle_connect 0x7fbf440355e0 2014-08-22 19:24:37,309.309 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.219446 7fbf427fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:37,312.312 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.222336 7fbf49d8e700 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:37,312.312 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.222549 7fbf49d8e700 1 librados: init done 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.222937 7fbf427fc700 10 client.4157.objecter ms_dispatch 0x7fbf44000f80 osd_map(55..55 src has 1..55) v3 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.222955 7fbf427fc700 3 client.4157.objecter handle_osd_map got epochs [55,55] > 0 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.222961 7fbf427fc700 3 client.4157.objecter handle_osd_map decoding full epoch 55 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.223144 7fbf427fc700 20 client.4157.objecter dump_active .. 0 homeless 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.223217 7fbf427fc700 10 client.4157.objecter ms_dispatch 0x7fbf44000f80 osd_map(55..55 src has 1..55) v3 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.223223 7fbf427fc700 3 client.4157.objecter handle_osd_map ignoring epochs [55,55] <= 55 2014-08-22 19:24:37,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.223226 7fbf427fc700 20 client.4157.objecter dump_active .. 0 homeless 2014-08-22 19:24:37,406.406 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.316493 7fbf49d8e700 20 client.4157.objecter shutdown clearing up homeless session... 2014-08-22 19:24:37,407.407 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:30.317422 7fbf49d8e700 1 librados: shutdown 2014-08-22 19:24:37,415.415 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003786', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001834', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:23:35.666841', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 874087720, u'health': u'HEALTH_OK', u'kb_used': 22693332, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:23:36.468081', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 877203152, u'health': u'HEALTH_OK', u'kb_used': 19577900, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:23:36.072261', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 874087712, u'health': u'HEALTH_OK', u'kb_used': 22693340, u'store_stats': {u'bytes_total': 2104367, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.c has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 42, u'data_bytes': 1230844437, u'bytes_used': 3327086592, u'version': 93, u'pgs_by_state': [{u'count': 42, u'state_name': u'active+clean'}], u'bytes_avail': 5994971959296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 55, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:24:37,416.416 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:24:37,417.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:37,417.417 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [2, 3, 4, 5, 0, 1] 2014-08-22 19:24:37,417.417 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-22 19:24:37,586.586 INFO:tasks.rados.rados.0.mira097.stdout:624: done (0 left) 2014-08-22 19:24:37,587.587 INFO:tasks.rados.rados.0.mira097.stdout:625: copy_from oid 349 from oid 218 current snap is 12 2014-08-22 19:24:37,591.591 INFO:tasks.rados.rados.0.mira097.stdout:626: write oid 471 current snap is 12 2014-08-22 19:24:37,591.591 INFO:tasks.rados.rados.0.mira097.stdout:626: seq_num 545 ranges {640946=645430,1790620=450326} 2014-08-22 19:24:37,622.622 INFO:tasks.rados.rados.0.mira097.stdout:626: writing mira09725123-471 from 640946 to 1286376 tid 1 2014-08-22 19:24:37,638.638 INFO:tasks.rados.rados.0.mira097.stdout:626: writing mira09725123-471 from 1790620 to 2240946 tid 2 2014-08-22 19:24:37,641.641 INFO:tasks.rados.rados.0.mira097.stdout:627: write oid 149 current snap is 12 2014-08-22 19:24:37,642.642 INFO:tasks.rados.rados.0.mira097.stdout:627: seq_num 546 ranges {702128=465810,1718256=724569,3221643=280485} 2014-08-22 19:24:37,658.658 INFO:tasks.rados.rados.0.mira097.stdout:627: writing mira09725123-149 from 702128 to 1167938 tid 1 2014-08-22 19:24:37,682.682 INFO:tasks.rados.rados.0.mira097.stdout:627: writing mira09725123-149 from 1718256 to 2442825 tid 2 2014-08-22 19:24:37,696.696 INFO:tasks.rados.rados.0.mira097.stdout:627: writing mira09725123-149 from 3221643 to 3502128 tid 3 2014-08-22 19:24:37,696.696 INFO:tasks.rados.rados.0.mira097.stdout:628: delete oid 52 current snap is 12 2014-08-22 19:24:37,696.696 INFO:tasks.rados.rados.0.mira097.stdout:625: finishing copy_from to mira09725123-349 2014-08-22 19:24:37,696.696 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 349 v 304 (ObjNum 217 snap 0 seq_num 217) dirty exists 2014-08-22 19:24:37,697.697 INFO:tasks.rados.rados.0.mira097.stdout:626: finishing write tid 1 to mira09725123-471 2014-08-22 19:24:37,697.697 INFO:tasks.rados.rados.0.mira097.stdout:626: finishing write tid 2 to mira09725123-471 2014-08-22 19:24:37,697.697 INFO:tasks.rados.rados.0.mira097.stdout:626: finishing write tid 4 to mira09725123-471 2014-08-22 19:24:37,697.697 INFO:tasks.rados.rados.0.mira097.stdout:626: finishing write tid 5 to mira09725123-471 2014-08-22 19:24:37,698.698 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 471 v 947 (ObjNum 545 snap 12 seq_num 545) dirty exists 2014-08-22 19:24:37,698.698 INFO:tasks.rados.rados.0.mira097.stdout:625: finishing copy_from racing read to mira09725123-349 2014-08-22 19:24:37,700.700 INFO:tasks.rados.rados.0.mira097.stdout:627: finishing write tid 1 to mira09725123-149 2014-08-22 19:24:37,700.700 INFO:tasks.rados.rados.0.mira097.stdout:627: finishing write tid 2 to mira09725123-149 2014-08-22 19:24:37,704.704 INFO:tasks.rados.rados.0.mira097.stdout:625: done (3 left) 2014-08-22 19:24:37,705.705 INFO:tasks.rados.rados.0.mira097.stdout:626: done (2 left) 2014-08-22 19:24:37,705.705 INFO:tasks.rados.rados.0.mira097.stdout:628: done (1 left) 2014-08-22 19:24:37,705.705 INFO:tasks.rados.rados.0.mira097.stdout:629: write oid 431 current snap is 12 2014-08-22 19:24:37,705.705 INFO:tasks.rados.rados.0.mira097.stdout:629: seq_num 548 ranges {589740=400000} 2014-08-22 19:24:37,727.727 INFO:tasks.rados.rados.0.mira097.stdout:629: writing mira09725123-431 from 589740 to 989740 tid 1 2014-08-22 19:24:37,727.727 INFO:tasks.rados.rados.0.mira097.stdout:630: snap_remove snap 9 2014-08-22 19:24:37,728.728 INFO:tasks.rados.rados.0.mira097.stdout:627: finishing write tid 3 to mira09725123-149 2014-08-22 19:24:37,729.729 INFO:tasks.rados.rados.0.mira097.stdout:627: finishing write tid 5 to mira09725123-149 2014-08-22 19:24:37,729.729 INFO:tasks.rados.rados.0.mira097.stdout:627: finishing write tid 6 to mira09725123-149 2014-08-22 19:24:37,729.729 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 149 v 783 (ObjNum 546 snap 12 seq_num 546) dirty exists 2014-08-22 19:24:37,740.740 INFO:tasks.rados.rados.0.mira097.stdout:629: finishing write tid 1 to mira09725123-431 2014-08-22 19:24:37,740.740 INFO:tasks.rados.rados.0.mira097.stdout:629: finishing write tid 3 to mira09725123-431 2014-08-22 19:24:37,753.753 INFO:tasks.rados.rados.0.mira097.stdout:629: finishing write tid 4 to mira09725123-431 2014-08-22 19:24:37,753.753 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 431 v 960 (ObjNum 548 snap 12 seq_num 548) dirty exists 2014-08-22 19:24:38,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.086913 7fe391bd4700 1 librados: starting msgr at :/0 2014-08-22 19:24:38,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.086920 7fe391bd4700 1 librados: starting objecter 2014-08-22 19:24:38,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.087479 7fe391bd4700 1 librados: setting wanted keys 2014-08-22 19:24:38,179.179 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.087487 7fe391bd4700 1 librados: calling monclient init 2014-08-22 19:24:38,179.179 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.089770 7fe38a7fc700 10 client.?.objecter ms_handle_connect 0x7fe38c0355e0 2014-08-22 19:24:38,180.180 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.089814 7fe38a7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:38,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.094101 7fe391bd4700 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:38,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.094235 7fe391bd4700 1 librados: init done 2014-08-22 19:24:38,185.185 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095113 7fe38a7fc700 10 client.4174.objecter ms_dispatch 0x7fe38c000f80 osd_map(56..56 src has 1..56) v3 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095133 7fe38a7fc700 3 client.4174.objecter handle_osd_map got epochs [56,56] > 0 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095139 7fe38a7fc700 3 client.4174.objecter handle_osd_map decoding full epoch 56 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095305 7fe38a7fc700 20 client.4174.objecter dump_active .. 0 homeless 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095390 7fe38a7fc700 10 client.4174.objecter ms_dispatch 0x7fe38c000f80 osd_map(56..56 src has 1..56) v3 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095411 7fe38a7fc700 3 client.4174.objecter handle_osd_map ignoring epochs [56,56] <= 56 2014-08-22 19:24:38,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.095418 7fe38a7fc700 20 client.4174.objecter dump_active .. 0 homeless 2014-08-22 19:24:38,596.596 INFO:tasks.rados.rados.0.mira097.stdout:627: done (2 left) 2014-08-22 19:24:38,597.597 INFO:tasks.rados.rados.0.mira097.stdout:629: done (1 left) 2014-08-22 19:24:38,597.597 INFO:tasks.rados.rados.0.mira097.stdout:630: done (0 left) 2014-08-22 19:24:38,597.597 INFO:tasks.rados.rados.0.mira097.stdout:631: copy_from oid 257 from oid 38 current snap is 12 2014-08-22 19:24:38,597.597 INFO:tasks.rados.rados.0.mira097.stdout:632: rollback oid 324 current snap is 12 2014-08-22 19:24:38,597.597 INFO:tasks.rados.rados.0.mira097.stdout:632: done (1 left) 2014-08-22 19:24:38,598.598 INFO:tasks.rados.rados.0.mira097.stdout:633: snap_create 2014-08-22 19:24:38,598.598 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:38,639.639 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.0. 2014-08-22 19:24:38,643.643 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.553079 7fe391bd4700 20 client.4174.objecter shutdown clearing up homeless session... 2014-08-22 19:24:38,643.643 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:31.553890 7fe391bd4700 1 librados: shutdown 2014-08-22 19:24:39,802.802 INFO:tasks.rados.rados.0.mira097.stdout:631: finishing copy_from to mira09725123-257 2014-08-22 19:24:39,803.803 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 257 v 57 (ObjNum 37 snap 0 seq_num 37) dirty exists 2014-08-22 19:24:39,803.803 INFO:tasks.rados.rados.0.mira097.stdout:631: finishing copy_from racing read to mira09725123-257 2014-08-22 19:24:39,803.803 INFO:tasks.rados.rados.0.mira097.stdout:631: done (0 left) 2014-08-22 19:24:43,100.100 INFO:tasks.rados.rados.0.mira097.stdout:633: done (0 left) 2014-08-22 19:24:43,101.101 INFO:tasks.rados.rados.0.mira097.stdout:634: copy_from oid 182 from oid 50 current snap is 13 2014-08-22 19:24:43,101.101 INFO:tasks.rados.rados.0.mira097.stdout:635: write oid 147 current snap is 13 2014-08-22 19:24:43,101.101 INFO:tasks.rados.rados.0.mira097.stdout:635: seq_num 549 ranges {685992=1} 2014-08-22 19:24:43,110.110 INFO:tasks.rados.rados.0.mira097.stdout:635: writing mira09725123-147 from 685992 to 685993 tid 1 2014-08-22 19:24:43,110.110 INFO:tasks.rados.rados.0.mira097.stdout:636: write oid 20 current snap is 13 2014-08-22 19:24:43,110.110 INFO:tasks.rados.rados.0.mira097.stdout:636: seq_num 550 ranges {497575=400000} 2014-08-22 19:24:43,133.133 INFO:tasks.rados.rados.0.mira097.stdout:636: writing mira09725123-20 from 497575 to 897575 tid 1 2014-08-22 19:24:43,133.133 INFO:tasks.rados.rados.0.mira097.stdout:637: delete oid 301 current snap is 13 2014-08-22 19:24:43,178.178 INFO:tasks.rados.rados.0.mira097.stdout:637: done (3 left) 2014-08-22 19:24:43,179.179 INFO:tasks.rados.rados.0.mira097.stdout:638: write oid 87 current snap is 13 2014-08-22 19:24:43,179.179 INFO:tasks.rados.rados.0.mira097.stdout:638: seq_num 552 ranges {787235=520252,2028010=359225} 2014-08-22 19:24:43,275.275 INFO:tasks.rados.rados.0.mira097.stdout:638: writing mira09725123-87 from 787235 to 1307487 tid 1 2014-08-22 19:24:43,275.275 INFO:tasks.rados.rados.0.mira097.stdout:638: writing mira09725123-87 from 2028010 to 2387235 tid 2 2014-08-22 19:24:43,275.275 INFO:tasks.rados.rados.0.mira097.stdout:639: snap_create 2014-08-22 19:24:43,275.275 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:635: finishing write tid 1 to mira09725123-147 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:635: finishing write tid 3 to mira09725123-147 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:635: finishing write tid 4 to mira09725123-147 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 147 v 881 (ObjNum 549 snap 13 seq_num 549) dirty exists 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:636: finishing write tid 1 to mira09725123-20 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:636: finishing write tid 3 to mira09725123-20 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:634: finishing copy_from to mira09725123-182 2014-08-22 19:24:43,276.276 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 182 v 55 (ObjNum 49 snap 0 seq_num 49) dirty exists 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:635: done (3 left) 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:636: finishing write tid 4 to mira09725123-20 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 20 v 967 (ObjNum 550 snap 13 seq_num 550) dirty exists 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:636: done (2 left) 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:634: finishing copy_from racing read to mira09725123-182 2014-08-22 19:24:43,277.277 INFO:tasks.rados.rados.0.mira097.stdout:634: done (1 left) 2014-08-22 19:24:43,278.278 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:43,652.652 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 1] out_osds: [0] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:43,652.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:43,652.652 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [2, 3, 4, 5, 1] 2014-08-22 19:24:43,652.652 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-08-22 19:24:44,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.032194 7f12464df700 1 librados: starting msgr at :/0 2014-08-22 19:24:44,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.032206 7f12464df700 1 librados: starting objecter 2014-08-22 19:24:44,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.032459 7f12464df700 1 librados: setting wanted keys 2014-08-22 19:24:44,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.032474 7f12464df700 1 librados: calling monclient init 2014-08-22 19:24:44,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.034196 7f123effd700 10 client.?.objecter ms_handle_connect 0x7f12400355e0 2014-08-22 19:24:44,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.034245 7f123effd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:44,594.594 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502042 7f12464df700 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:44,594.594 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502230 7f12464df700 1 librados: init done 2014-08-22 19:24:44,594.594 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502519 7f123effd700 10 client.4158.objecter ms_dispatch 0x7f1240000f80 osd_map(59..59 src has 1..59) v3 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502527 7f123effd700 3 client.4158.objecter handle_osd_map got epochs [59,59] > 0 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502531 7f123effd700 3 client.4158.objecter handle_osd_map decoding full epoch 59 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502616 7f123effd700 20 client.4158.objecter dump_active .. 0 homeless 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502701 7f123effd700 10 client.4158.objecter ms_dispatch 0x7f1240000f80 osd_map(59..59 src has 1..59) v3 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502706 7f123effd700 3 client.4158.objecter handle_osd_map ignoring epochs [59,59] <= 59 2014-08-22 19:24:44,595.595 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:37.502709 7f123effd700 20 client.4158.objecter dump_active .. 0 homeless 2014-08-22 19:24:45,690.690 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.2. 2014-08-22 19:24:45,701.701 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:38.609892 7f12464df700 20 client.4158.objecter shutdown clearing up homeless session... 2014-08-22 19:24:45,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:38.611813 7f12464df700 1 librados: shutdown 2014-08-22 19:24:50,716.716 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 1] out_osds: [0, 2] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:50,716.716 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:50,716.716 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:24:50,716.716 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:51,195.195 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.103897 7fcd05e6e700 1 librados: starting msgr at :/0 2014-08-22 19:24:51,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.103910 7fcd05e6e700 1 librados: starting objecter 2014-08-22 19:24:51,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.104251 7fcd05e6e700 1 librados: setting wanted keys 2014-08-22 19:24:51,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.104257 7fcd05e6e700 1 librados: calling monclient init 2014-08-22 19:24:51,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.105831 7fccfe7fc700 10 client.?.objecter ms_handle_connect 0x7fcd000355e0 2014-08-22 19:24:51,197.197 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.105872 7fccfe7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:51,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.108820 7fcd05e6e700 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:51,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.108969 7fcd05e6e700 1 librados: init done 2014-08-22 19:24:51,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109553 7fccfe7fc700 10 client.4161.objecter ms_dispatch 0x7fcd00000f80 osd_map(61..61 src has 1..61) v3 2014-08-22 19:24:51,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109566 7fccfe7fc700 3 client.4161.objecter handle_osd_map got epochs [61,61] > 0 2014-08-22 19:24:51,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109569 7fccfe7fc700 3 client.4161.objecter handle_osd_map decoding full epoch 61 2014-08-22 19:24:51,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109736 7fccfe7fc700 20 client.4161.objecter dump_active .. 0 homeless 2014-08-22 19:24:51,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109796 7fccfe7fc700 10 client.4161.objecter ms_dispatch 0x7fcd00000f80 osd_map(61..61 src has 1..61) v3 2014-08-22 19:24:51,202.202 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109801 7fccfe7fc700 3 client.4161.objecter handle_osd_map ignoring epochs [61,61] <= 61 2014-08-22 19:24:51,202.202 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.109803 7fccfe7fc700 20 client.4161.objecter dump_active .. 0 homeless 2014-08-22 19:24:51,305.305 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:51,654.654 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.217492 7fcd05e6e700 20 client.4161.objecter shutdown clearing up homeless session... 2014-08-22 19:24:51,654.654 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:44.217815 7fcd05e6e700 1 librados: shutdown 2014-08-22 19:24:51,657.657 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:24:51,658.658 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 44' 2014-08-22 19:24:52,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.109306 7fe00a4e6700 1 librados: starting msgr at :/0 2014-08-22 19:24:52,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.109313 7fe00a4e6700 1 librados: starting objecter 2014-08-22 19:24:52,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.109582 7fe00a4e6700 1 librados: setting wanted keys 2014-08-22 19:24:52,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.109593 7fe00a4e6700 1 librados: calling monclient init 2014-08-22 19:24:52,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.111284 7fe002ffd700 10 client.?.objecter ms_handle_connect 0x7fe0040355e0 2014-08-22 19:24:52,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.111336 7fe002ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:52,204.204 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.114286 7fe00a4e6700 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:52,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.114459 7fe00a4e6700 1 librados: init done 2014-08-22 19:24:52,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115036 7fe002ffd700 10 client.4164.objecter ms_dispatch 0x7fe004000f80 osd_map(61..61 src has 1..61) v3 2014-08-22 19:24:52,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115048 7fe002ffd700 3 client.4164.objecter handle_osd_map got epochs [61,61] > 0 2014-08-22 19:24:52,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115051 7fe002ffd700 3 client.4164.objecter handle_osd_map decoding full epoch 61 2014-08-22 19:24:52,205.205 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115179 7fe002ffd700 20 client.4164.objecter dump_active .. 0 homeless 2014-08-22 19:24:52,206.206 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115237 7fe002ffd700 10 client.4164.objecter ms_dispatch 0x7fe004000f80 osd_map(61..61 src has 1..61) v3 2014-08-22 19:24:52,206.206 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115242 7fe002ffd700 3 client.4164.objecter handle_osd_map ignoring epochs [61,61] <= 61 2014-08-22 19:24:52,206.206 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.115244 7fe002ffd700 20 client.4164.objecter dump_active .. 0 homeless 2014-08-22 19:24:52,533.533 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 44 2014-08-22 19:24:52,534.534 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.420448 7fe00a4e6700 20 client.4164.objecter shutdown clearing up homeless session... 2014-08-22 19:24:52,534.534 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:45.420875 7fe00a4e6700 1 librados: shutdown 2014-08-22 19:24:55,142.142 INFO:tasks.rados.rados.0.mira097.stdout:638: finishing write tid 1 to mira09725123-87 2014-08-22 19:24:55,143.143 INFO:tasks.rados.rados.0.mira097.stdout:638: finishing write tid 2 to mira09725123-87 2014-08-22 19:24:55,144.144 INFO:tasks.rados.rados.0.mira097.stdout:638: finishing write tid 4 to mira09725123-87 2014-08-22 19:24:55,144.144 INFO:tasks.rados.rados.0.mira097.stdout:638: finishing write tid 5 to mira09725123-87 2014-08-22 19:24:55,145.145 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 87 v 792 (ObjNum 552 snap 13 seq_num 552) dirty exists 2014-08-22 19:24:55,145.145 INFO:tasks.rados.rados.0.mira097.stdout:638: done (0 left) 2014-08-22 19:24:55,514.514 INFO:tasks.rados.rados.0.mira097.stdout:639: done (0 left) 2014-08-22 19:24:55,514.514 INFO:tasks.rados.rados.0.mira097.stdout:640: snap_create 2014-08-22 19:24:56,596.596 INFO:tasks.rados.rados.0.mira097.stdout:640: done (0 left) 2014-08-22 19:24:56,597.597 INFO:tasks.rados.rados.0.mira097.stdout:641: write oid 490 current snap is 15 2014-08-22 19:24:56,597.597 INFO:tasks.rados.rados.0.mira097.stdout:641: seq_num 553 ranges {482464=643481,1892542=189922} 2014-08-22 19:24:56,627.627 INFO:tasks.rados.rados.0.mira097.stdout:641: writing mira09725123-490 from 482464 to 1125945 tid 1 2014-08-22 19:24:56,632.632 INFO:tasks.rados.rados.0.mira097.stdout:641: writing mira09725123-490 from 1892542 to 2082464 tid 2 2014-08-22 19:24:56,634.634 INFO:tasks.rados.rados.0.mira097.stdout:642: copy_from oid 247 from oid 399 current snap is 15 2014-08-22 19:24:56,634.634 INFO:tasks.rados.rados.0.mira097.stdout:643: write oid 486 current snap is 15 2014-08-22 19:24:56,634.634 INFO:tasks.rados.rados.0.mira097.stdout:643: seq_num 554 ranges {427294=615285,1820178=671603,2827293=1} 2014-08-22 19:24:56,657.657 INFO:tasks.rados.rados.0.mira097.stdout:643: writing mira09725123-486 from 427294 to 1042579 tid 1 2014-08-22 19:24:56,688.688 INFO:tasks.rados.rados.0.mira097.stdout:643: writing mira09725123-486 from 1820178 to 2491781 tid 2 2014-08-22 19:24:56,688.688 INFO:tasks.rados.rados.0.mira097.stdout:643: writing mira09725123-486 from 2827293 to 2827294 tid 3 2014-08-22 19:24:56,689.689 INFO:tasks.rados.rados.0.mira097.stdout:644: snap_create 2014-08-22 19:24:56,689.689 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:24:56,709.709 INFO:tasks.rados.rados.0.mira097.stdout:643: finishing write tid 1 to mira09725123-486 2014-08-22 19:24:56,767.767 INFO:tasks.rados.rados.0.mira097.stdout:643: finishing write tid 2 to mira09725123-486 2014-08-22 19:24:56,804.804 INFO:tasks.rados.rados.0.mira097.stdout:643: finishing write tid 3 to mira09725123-486 2014-08-22 19:24:56,804.804 INFO:tasks.rados.rados.0.mira097.stdout:643: finishing write tid 5 to mira09725123-486 2014-08-22 19:24:56,805.805 INFO:tasks.rados.rados.0.mira097.stdout:642: finishing copy_from to mira09725123-247 2014-08-22 19:24:56,805.805 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 247 v 552 (ObjNum 398 snap 0 seq_num 398) dirty exists 2014-08-22 19:24:56,817.817 INFO:tasks.rados.rados.0.mira097.stdout:641: finishing write tid 1 to mira09725123-490 2014-08-22 19:24:56,817.817 INFO:tasks.rados.rados.0.mira097.stdout:641: finishing write tid 2 to mira09725123-490 2014-08-22 19:24:56,833.833 INFO:tasks.rados.rados.0.mira097.stdout:642: finishing copy_from racing read to mira09725123-247 2014-08-22 19:24:56,836.836 INFO:tasks.rados.rados.0.mira097.stdout:642: done (2 left) 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout:641: finishing write tid 4 to mira09725123-490 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout:641: finishing write tid 5 to mira09725123-490 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 490 v 978 (ObjNum 553 snap 15 seq_num 553) dirty exists 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout:641: done (1 left) 2014-08-22 19:24:56,837.837 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:24:56,843.843 INFO:tasks.rados.rados.0.mira097.stdout:643: finishing write tid 6 to mira09725123-486 2014-08-22 19:24:56,843.843 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 486 v 801 (ObjNum 554 snap 15 seq_num 554) dirty exists 2014-08-22 19:24:56,843.843 INFO:tasks.rados.rados.0.mira097.stdout:643: done (0 left) 2014-08-22 19:24:57,536.536 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 1] out_osds: [0, 2] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:24:57,536.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:24:57,536.536 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:24:57,537.537 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:24:57,621.621 INFO:tasks.rados.rados.0.mira097.stdout:644: done (0 left) 2014-08-22 19:24:57,621.621 INFO:tasks.rados.rados.0.mira097.stdout:645: read oid 177 snap -1 2014-08-22 19:24:57,622.622 INFO:tasks.rados.rados.0.mira097.stdout:646: delete oid 72 current snap is 16 2014-08-22 19:24:57,707.707 INFO:tasks.rados.rados.0.mira097.stdout:645: expect (ObjNum 176 snap 0 seq_num 176) 2014-08-22 19:24:57,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.895925 7f5f8d6f9700 1 librados: starting msgr at :/0 2014-08-22 19:24:57,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.895929 7f5f8d6f9700 1 librados: starting objecter 2014-08-22 19:24:57,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.896053 7f5f8d6f9700 1 librados: setting wanted keys 2014-08-22 19:24:57,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.896061 7f5f8d6f9700 1 librados: calling monclient init 2014-08-22 19:24:57,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.896822 7f5f8658b700 10 client.?.objecter ms_handle_connect 0x7f5f880355e0 2014-08-22 19:24:57,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.896846 7f5f8658b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:24:57,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.898565 7f5f8d6f9700 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:24:57,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.898679 7f5f8d6f9700 1 librados: init done 2014-08-22 19:24:57,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899058 7f5f8658b700 10 client.4160.objecter ms_dispatch 0x7f5f88000f80 osd_map(66..66 src has 1..66) v3 2014-08-22 19:24:57,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899064 7f5f8658b700 3 client.4160.objecter handle_osd_map got epochs [66,66] > 0 2014-08-22 19:24:57,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899066 7f5f8658b700 3 client.4160.objecter handle_osd_map decoding full epoch 66 2014-08-22 19:24:57,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899134 7f5f8658b700 20 client.4160.objecter dump_active .. 0 homeless 2014-08-22 19:24:57,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899161 7f5f8658b700 10 client.4160.objecter ms_dispatch 0x7f5f88000f80 osd_map(66..66 src has 1..66) v3 2014-08-22 19:24:57,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899163 7f5f8658b700 3 client.4160.objecter handle_osd_map ignoring epochs [66,66] <= 66 2014-08-22 19:24:57,991.991 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.899164 7f5f8658b700 20 client.4160.objecter dump_active .. 0 homeless 2014-08-22 19:24:58,169.169 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:24:58,507.507 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.981495 7f5f8d6f9700 20 client.4160.objecter shutdown clearing up homeless session... 2014-08-22 19:24:58,507.507 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:50.981891 7f5f8d6f9700 1 librados: shutdown 2014-08-22 19:24:58,595.595 INFO:tasks.rados.rados.0.mira097.stdout:645: done (1 left) 2014-08-22 19:24:58,595.595 INFO:tasks.rados.rados.0.mira097.stdout:646: done (0 left) 2014-08-22 19:24:58,595.595 INFO:tasks.rados.rados.0.mira097.stdout:647: write oid 139 current snap is 16 2014-08-22 19:24:58,596.596 INFO:tasks.rados.rados.0.mira097.stdout:647: seq_num 556 ranges {582682=443566,1495995=714841,2758379=577829,3782681=1} 2014-08-22 19:24:58,622.622 INFO:tasks.rados.rados.0.mira097.stdout:647: writing mira09725123-139 from 582682 to 1026248 tid 1 2014-08-22 19:24:58,647.647 INFO:tasks.rados.rados.0.mira097.stdout:647: writing mira09725123-139 from 1495995 to 2210836 tid 2 2014-08-22 19:24:58,669.669 INFO:tasks.rados.rados.0.mira097.stdout:647: writing mira09725123-139 from 2758379 to 3336208 tid 3 2014-08-22 19:24:58,673.673 INFO:tasks.rados.rados.0.mira097.stdout:647: writing mira09725123-139 from 3782681 to 3782682 tid 4 2014-08-22 19:24:58,673.673 INFO:tasks.rados.rados.0.mira097.stdout:648: read oid 128 snap -1 2014-08-22 19:24:58,673.673 INFO:tasks.rados.rados.0.mira097.stdout:649: snap_create 2014-08-22 19:24:58,673.673 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:24:58,717.717 INFO:tasks.rados.rados.0.mira097.stdout:648: expect (ObjNum 127 snap 0 seq_num 127) 2014-08-22 19:24:59,633.633 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 1 to mira09725123-139 2014-08-22 19:24:59,633.633 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 2 to mira09725123-139 2014-08-22 19:24:59,634.634 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 3 to mira09725123-139 2014-08-22 19:24:59,634.634 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 4 to mira09725123-139 2014-08-22 19:24:59,634.634 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 6 to mira09725123-139 2014-08-22 19:24:59,634.634 INFO:tasks.rados.rados.0.mira097.stdout:647: finishing write tid 7 to mira09725123-139 2014-08-22 19:24:59,634.634 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 139 v 888 (ObjNum 556 snap 16 seq_num 556) dirty exists 2014-08-22 19:24:59,635.635 INFO:tasks.rados.rados.0.mira097.stdout:647: done (1 left) 2014-08-22 19:24:59,635.635 INFO:tasks.rados.rados.0.mira097.stdout:648: done (0 left) 2014-08-22 19:25:00,679.679 INFO:tasks.rados.rados.0.mira097.stdout:649: done (0 left) 2014-08-22 19:25:00,679.679 INFO:tasks.rados.rados.0.mira097.stdout:650: rollback oid 402 current snap is 17 2014-08-22 19:25:00,679.679 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 402 to 15 2014-08-22 19:25:00,679.679 INFO:tasks.rados.rados.0.mira097.stdout:651: snap_remove snap 12 2014-08-22 19:25:00,690.690 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 402 v 891 (ObjNum 401 snap 0 seq_num 401) dirty exists 2014-08-22 19:25:01,683.683 INFO:tasks.rados.rados.0.mira097.stdout:650: done (1 left) 2014-08-22 19:25:01,683.683 INFO:tasks.rados.rados.0.mira097.stdout:651: done (0 left) 2014-08-22 19:25:01,683.683 INFO:tasks.rados.rados.0.mira097.stdout:652: write oid 84 current snap is 17 2014-08-22 19:25:01,683.683 INFO:tasks.rados.rados.0.mira097.stdout:652: seq_num 557 ranges {527510=1} 2014-08-22 19:25:01,693.693 INFO:tasks.rados.rados.0.mira097.stdout:652: writing mira09725123-84 from 527510 to 527511 tid 1 2014-08-22 19:25:01,694.694 INFO:tasks.rados.rados.0.mira097.stdout:653: delete oid 57 current snap is 17 2014-08-22 19:25:01,723.723 INFO:tasks.rados.rados.0.mira097.stdout:652: finishing write tid 1 to mira09725123-84 2014-08-22 19:25:01,725.725 INFO:tasks.rados.rados.0.mira097.stdout:652: finishing write tid 3 to mira09725123-84 2014-08-22 19:25:01,728.728 INFO:tasks.rados.rados.0.mira097.stdout:652: finishing write tid 4 to mira09725123-84 2014-08-22 19:25:01,728.728 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 84 v 988 (ObjNum 557 snap 17 seq_num 557) dirty exists 2014-08-22 19:25:01,756.756 INFO:tasks.rados.rados.0.mira097.stdout:652: done (1 left) 2014-08-22 19:25:01,756.756 INFO:tasks.rados.rados.0.mira097.stdout:653: done (0 left) 2014-08-22 19:25:01,757.757 INFO:tasks.rados.rados.0.mira097.stdout:654: snap_create 2014-08-22 19:25:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:654: done (0 left) 2014-08-22 19:25:02,697.697 INFO:tasks.rados.rados.0.mira097.stdout:655: snap_create 2014-08-22 19:25:03,508.508 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 1] out_osds: [0, 2] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:03,508.508 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:03,508.508 INFO:tasks.thrashosds.thrasher:Adding osd 2 2014-08-22 19:25:03,509.509 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-08-22 19:25:03,704.704 INFO:tasks.rados.rados.0.mira097.stdout:655: done (0 left) 2014-08-22 19:25:03,704.704 INFO:tasks.rados.rados.0.mira097.stdout:656: write oid 226 current snap is 19 2014-08-22 19:25:03,704.704 INFO:tasks.rados.rados.0.mira097.stdout:656: seq_num 559 ranges {683923=546499,1483922=1} 2014-08-22 19:25:03,730.730 INFO:tasks.rados.rados.0.mira097.stdout:656: writing mira09725123-226 from 683923 to 1230422 tid 1 2014-08-22 19:25:03,733.733 INFO:tasks.rados.rados.0.mira097.stdout:656: writing mira09725123-226 from 1483922 to 1483923 tid 2 2014-08-22 19:25:03,734.734 INFO:tasks.rados.rados.0.mira097.stdout:657: delete oid 235 current snap is 19 2014-08-22 19:25:03,983.983 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.889962 7f71d53e3700 1 librados: starting msgr at :/0 2014-08-22 19:25:03,984.984 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.889973 7f71d53e3700 1 librados: starting objecter 2014-08-22 19:25:03,984.984 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.890239 7f71d53e3700 1 librados: setting wanted keys 2014-08-22 19:25:03,984.984 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.890254 7f71d53e3700 1 librados: calling monclient init 2014-08-22 19:25:03,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.892087 7f71ce2e8700 10 client.?.objecter ms_handle_connect 0x7f71d00355e0 2014-08-22 19:25:03,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.892157 7f71ce2e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:03,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895175 7f71d53e3700 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:03,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895376 7f71d53e3700 1 librados: init done 2014-08-22 19:25:04,062.062 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895779 7f71ce2e8700 10 client.4167.objecter ms_dispatch 0x7f71d0000f80 osd_map(70..70 src has 1..70) v3 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895800 7f71ce2e8700 3 client.4167.objecter handle_osd_map got epochs [70,70] > 0 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895804 7f71ce2e8700 3 client.4167.objecter handle_osd_map decoding full epoch 70 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.895984 7f71ce2e8700 20 client.4167.objecter dump_active .. 0 homeless 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.896031 7f71ce2e8700 10 client.4167.objecter ms_dispatch 0x7f71d0000f80 osd_map(70..70 src has 1..70) v3 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.896035 7f71ce2e8700 3 client.4167.objecter handle_osd_map ignoring epochs [70,70] <= 70 2014-08-22 19:25:04,063.063 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:56.896037 7f71ce2e8700 20 client.4167.objecter dump_active .. 0 homeless 2014-08-22 19:25:04,063.063 INFO:tasks.rados.rados.0.mira097.stdout:657: done (1 left) 2014-08-22 19:25:04,064.064 INFO:tasks.rados.rados.0.mira097.stdout:658: delete oid 301 current snap is 19 2014-08-22 19:25:04,616.616 INFO:tasks.rados.rados.0.mira097.stdout:658: done (1 left) 2014-08-22 19:25:04,616.616 INFO:tasks.rados.rados.0.mira097.stdout:659: copy_from oid 426 from oid 170 current snap is 19 2014-08-22 19:25:04,617.617 INFO:tasks.rados.rados.0.mira097.stdout:660: snap_remove snap 18 2014-08-22 19:25:04,626.626 INFO:tasks.rados.rados.0.mira097.stdout:659: finishing copy_from to mira09725123-426 2014-08-22 19:25:04,627.627 INFO:tasks.rados.rados.0.mira097.stdout:659: got expected ENOENT (src dne) 2014-08-22 19:25:04,627.627 INFO:tasks.rados.rados.0.mira097.stdout:659: finishing copy_from racing read to mira09725123-426 2014-08-22 19:25:04,718.718 INFO:tasks.rados.rados.0.mira097.stdout:659: done (2 left) 2014-08-22 19:25:04,718.718 INFO:tasks.rados.rados.0.mira097.stdout:660: done (1 left) 2014-08-22 19:25:04,718.718 INFO:tasks.rados.rados.0.mira097.stdout:661: snap_create 2014-08-22 19:25:04,718.718 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:04,741.741 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.2. 2014-08-22 19:25:04,746.746 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:57.655993 7f71d53e3700 20 client.4167.objecter shutdown clearing up homeless session... 2014-08-22 19:25:04,746.746 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:24:57.656747 7f71d53e3700 1 librados: shutdown 2014-08-22 19:25:04,755.755 INFO:tasks.thrashosds.thrasher:Added osd 2 2014-08-22 19:25:05,810.810 INFO:tasks.rados.rados.0.mira097.stdout:656: finishing write tid 1 to mira09725123-226 2014-08-22 19:25:05,811.811 INFO:tasks.rados.rados.0.mira097.stdout:656: finishing write tid 2 to mira09725123-226 2014-08-22 19:25:05,811.811 INFO:tasks.rados.rados.0.mira097.stdout:656: finishing write tid 4 to mira09725123-226 2014-08-22 19:25:05,814.814 INFO:tasks.rados.rados.0.mira097.stdout:656: finishing write tid 5 to mira09725123-226 2014-08-22 19:25:05,814.814 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 226 v 816 (ObjNum 559 snap 19 seq_num 559) dirty exists 2014-08-22 19:25:05,815.815 INFO:tasks.rados.rados.0.mira097.stdout:656: done (0 left) 2014-08-22 19:25:07,867.867 INFO:tasks.rados.rados.0.mira097.stdout:661: done (0 left) 2014-08-22 19:25:07,868.868 INFO:tasks.rados.rados.0.mira097.stdout:662: write oid 52 current snap is 20 2014-08-22 19:25:07,868.868 INFO:tasks.rados.rados.0.mira097.stdout:662: seq_num 562 ranges {668812=615384,2075571=593241} 2014-08-22 19:25:07,897.897 INFO:tasks.rados.rados.0.mira097.stdout:662: writing mira09725123-52 from 668812 to 1284196 tid 1 2014-08-22 19:25:07,922.922 INFO:tasks.rados.rados.0.mira097.stdout:662: writing mira09725123-52 from 2075571 to 2668812 tid 2 2014-08-22 19:25:07,925.925 INFO:tasks.rados.rados.0.mira097.stdout:663: write oid 423 current snap is 20 2014-08-22 19:25:07,925.925 INFO:tasks.rados.rados.0.mira097.stdout:663: seq_num 563 ranges {612618=622260,1672327=592703,2949803=462815} 2014-08-22 19:25:07,945.945 INFO:tasks.rados.rados.0.mira097.stdout:663: writing mira09725123-423 from 612618 to 1234878 tid 1 2014-08-22 19:25:07,965.965 INFO:tasks.rados.rados.0.mira097.stdout:663: writing mira09725123-423 from 1672327 to 2265030 tid 2 2014-08-22 19:25:07,984.984 INFO:tasks.rados.rados.0.mira097.stdout:663: writing mira09725123-423 from 2949803 to 3412618 tid 3 2014-08-22 19:25:07,987.987 INFO:tasks.rados.rados.0.mira097.stdout:664: read oid 44 snap -1 2014-08-22 19:25:07,988.988 INFO:tasks.rados.rados.0.mira097.stdout:662: finishing write tid 1 to mira09725123-52 2014-08-22 19:25:07,988.988 INFO:tasks.rados.rados.0.mira097.stdout:662: finishing write tid 2 to mira09725123-52 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:662: finishing write tid 4 to mira09725123-52 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:662: finishing write tid 5 to mira09725123-52 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 52 v 998 (ObjNum 562 snap 20 seq_num 562) dirty exists 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:663: finishing write tid 1 to mira09725123-423 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:663: finishing write tid 2 to mira09725123-423 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:662: done (2 left) 2014-08-22 19:25:07,989.989 INFO:tasks.rados.rados.0.mira097.stdout:665: write oid 312 current snap is 20 2014-08-22 19:25:07,990.990 INFO:tasks.rados.rados.0.mira097.stdout:665: seq_num 564 ranges {424199=594064,1466714=662898,2714931=507865,3624198=1} 2014-08-22 19:25:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:665: writing mira09725123-312 from 424199 to 1018263 tid 1 2014-08-22 19:25:08,039.039 INFO:tasks.rados.rados.0.mira097.stdout:665: writing mira09725123-312 from 1466714 to 2129612 tid 2 2014-08-22 19:25:08,047.047 INFO:tasks.rados.rados.0.mira097.stdout:665: writing mira09725123-312 from 2714931 to 3222796 tid 3 2014-08-22 19:25:08,097.097 INFO:tasks.rados.rados.0.mira097.stdout:665: writing mira09725123-312 from 3624198 to 3624199 tid 4 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:666: delete oid 138 current snap is 20 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:663: finishing write tid 3 to mira09725123-423 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:663: finishing write tid 5 to mira09725123-423 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:663: finishing write tid 6 to mira09725123-423 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 423 v 1003 (ObjNum 563 snap 20 seq_num 563) dirty exists 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 1 to mira09725123-312 2014-08-22 19:25:08,098.098 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 2 to mira09725123-312 2014-08-22 19:25:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 3 to mira09725123-312 2014-08-22 19:25:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 4 to mira09725123-312 2014-08-22 19:25:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 6 to mira09725123-312 2014-08-22 19:25:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:665: finishing write tid 7 to mira09725123-312 2014-08-22 19:25:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 312 v 980 (ObjNum 564 snap 20 seq_num 564) dirty exists 2014-08-22 19:25:08,105.105 INFO:tasks.rados.rados.0.mira097.stdout:663: done (3 left) 2014-08-22 19:25:08,105.105 INFO:tasks.rados.rados.0.mira097.stdout:665: done (2 left) 2014-08-22 19:25:08,105.105 INFO:tasks.rados.rados.0.mira097.stdout:666: done (1 left) 2014-08-22 19:25:08,105.105 INFO:tasks.rados.rados.0.mira097.stdout:667: snap_remove snap 17 2014-08-22 19:25:09,756.756 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 1, 2] out_osds: [0] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:09,756.756 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:09,756.756 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [3, 4, 5, 1, 2] 2014-08-22 19:25:09,757.757 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 4' 2014-08-22 19:25:09,841.841 INFO:tasks.rados.rados.0.mira097.stdout:664: expect (ObjNum 43 snap 0 seq_num 43) 2014-08-22 19:25:10,020.020 INFO:tasks.rados.rados.0.mira097.stdout:664: done (1 left) 2014-08-22 19:25:10,021.021 INFO:tasks.rados.rados.0.mira097.stdout:667: done (0 left) 2014-08-22 19:25:10,021.021 INFO:tasks.rados.rados.0.mira097.stdout:668: read oid 217 snap -1 2014-08-22 19:25:10,021.021 INFO:tasks.rados.rados.0.mira097.stdout:669: copy_from oid 342 from oid 116 current snap is 20 2014-08-22 19:25:10,021.021 INFO:tasks.rados.rados.0.mira097.stdout:670: read oid 30 snap -1 2014-08-22 19:25:10,024.024 INFO:tasks.rados.rados.0.mira097.stdout:671: read oid 350 snap 14 2014-08-22 19:25:10,024.024 INFO:tasks.rados.rados.0.mira097.stdout:672: copy_from oid 388 from oid 389 current snap is 20 2014-08-22 19:25:10,024.024 INFO:tasks.rados.rados.0.mira097.stdout:673: read oid 93 snap -1 2014-08-22 19:25:10,025.025 INFO:tasks.rados.rados.0.mira097.stdout:674: write oid 78 current snap is 20 2014-08-22 19:25:10,025.025 INFO:tasks.rados.rados.0.mira097.stdout:674: seq_num 566 ranges {712835=400000} 2014-08-22 19:25:10,046.046 INFO:tasks.rados.rados.0.mira097.stdout:674: writing mira09725123-78 from 712835 to 1112835 tid 1 2014-08-22 19:25:10,046.046 INFO:tasks.rados.rados.0.mira097.stdout:675: copy_from oid 85 from oid 296 current snap is 20 2014-08-22 19:25:10,046.046 INFO:tasks.rados.rados.0.mira097.stdout:676: write oid 450 current snap is 20 2014-08-22 19:25:10,046.046 INFO:tasks.rados.rados.0.mira097.stdout:676: seq_num 567 ranges {409089=400000} 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:676: writing mira09725123-450 from 409089 to 809089 tid 1 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:677: write oid 255 current snap is 20 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:677: seq_num 568 ranges {752894=517378,1552893=1} 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:677: writing mira09725123-255 from 752894 to 1270272 tid 1 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:677: writing mira09725123-255 from 1552893 to 1552894 tid 2 2014-08-22 19:25:10,104.104 INFO:tasks.rados.rados.0.mira097.stdout:678: snap_create 2014-08-22 19:25:10,105.105 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 10 2014-08-22 19:25:10,105.105 INFO:tasks.rados.rados.0.mira097.stdout:668: expect (ObjNum 216 snap 0 seq_num 216) 2014-08-22 19:25:10,236.236 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.144689 7fa1e1182700 1 librados: starting msgr at :/0 2014-08-22 19:25:10,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.144697 7fa1e1182700 1 librados: starting objecter 2014-08-22 19:25:10,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.144818 7fa1e1182700 1 librados: setting wanted keys 2014-08-22 19:25:10,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.144824 7fa1e1182700 1 librados: calling monclient init 2014-08-22 19:25:10,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.145537 7fa1da0ae700 10 client.?.objecter ms_handle_connect 0x7fa1dc0355e0 2014-08-22 19:25:10,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.145556 7fa1da0ae700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:10,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.146883 7fa1e1182700 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:10,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.146987 7fa1e1182700 1 librados: init done 2014-08-22 19:25:10,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147500 7fa1da0ae700 10 client.4163.objecter ms_dispatch 0x7fa1dc000f80 osd_map(74..74 src has 1..74) v3 2014-08-22 19:25:10,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147506 7fa1da0ae700 3 client.4163.objecter handle_osd_map got epochs [74,74] > 0 2014-08-22 19:25:10,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147508 7fa1da0ae700 3 client.4163.objecter handle_osd_map decoding full epoch 74 2014-08-22 19:25:10,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147576 7fa1da0ae700 20 client.4163.objecter dump_active .. 0 homeless 2014-08-22 19:25:10,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147601 7fa1da0ae700 10 client.4163.objecter ms_dispatch 0x7fa1dc000f80 osd_map(74..74 src has 1..74) v3 2014-08-22 19:25:10,243.243 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147605 7fa1da0ae700 3 client.4163.objecter handle_osd_map ignoring epochs [74,74] <= 74 2014-08-22 19:25:10,243.243 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:03.147606 7fa1da0ae700 20 client.4163.objecter dump_active .. 0 homeless 2014-08-22 19:25:10,466.466 INFO:tasks.rados.rados.0.mira097.stdout:676: finishing write tid 1 to mira09725123-450 2014-08-22 19:25:10,466.466 INFO:tasks.rados.rados.0.mira097.stdout:676: finishing write tid 3 to mira09725123-450 2014-08-22 19:25:10,466.466 INFO:tasks.rados.rados.0.mira097.stdout:676: finishing write tid 4 to mira09725123-450 2014-08-22 19:25:10,466.466 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 450 v 995 (ObjNum 567 snap 20 seq_num 567) dirty exists 2014-08-22 19:25:10,466.466 INFO:tasks.rados.rados.0.mira097.stdout:670: expect (ObjNum 29 snap 0 seq_num 29) 2014-08-22 19:25:10,983.983 INFO:tasks.rados.rados.0.mira097.stdout:677: finishing write tid 1 to mira09725123-255 2014-08-22 19:25:10,984.984 INFO:tasks.rados.rados.0.mira097.stdout:677: finishing write tid 2 to mira09725123-255 2014-08-22 19:25:10,984.984 INFO:tasks.rados.rados.0.mira097.stdout:677: finishing write tid 4 to mira09725123-255 2014-08-22 19:25:10,984.984 INFO:tasks.rados.rados.0.mira097.stdout:671: expect (ObjNum 349 snap 0 seq_num 349) 2014-08-22 19:25:12,311.311 INFO:tasks.rados.rados.0.mira097.stdout:669: finishing copy_from to mira09725123-342 2014-08-22 19:25:12,311.311 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 342 v 141 (ObjNum 115 snap 0 seq_num 115) dirty exists 2014-08-22 19:25:12,311.311 INFO:tasks.rados.rados.0.mira097.stdout:672: finishing copy_from to mira09725123-388 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 388 v 535 (ObjNum 388 snap 0 seq_num 388) dirty exists 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:672: finishing copy_from racing read to mira09725123-388 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:675: finishing copy_from to mira09725123-85 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 85 v 392 (ObjNum 295 snap 0 seq_num 295) dirty exists 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:675: finishing copy_from racing read to mira09725123-85 2014-08-22 19:25:12,312.312 INFO:tasks.rados.rados.0.mira097.stdout:673: expect (ObjNum 92 snap 0 seq_num 92) 2014-08-22 19:25:12,694.694 INFO:tasks.rados.rados.0.mira097.stdout:677: finishing write tid 5 to mira09725123-255 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 255 v 830 (ObjNum 568 snap 20 seq_num 568) dirty exists 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:669: finishing copy_from racing read to mira09725123-342 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:674: finishing write tid 1 to mira09725123-78 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:668: done (9 left) 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:669: done (8 left) 2014-08-22 19:25:12,695.695 INFO:tasks.rados.rados.0.mira097.stdout:670: done (7 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:671: done (6 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:672: done (5 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:673: done (4 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:675: done (3 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:676: done (2 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout:677: done (1 left) 2014-08-22 19:25:12,696.696 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:12,697.697 INFO:tasks.rados.rados.0.mira097.stdout:674: finishing write tid 3 to mira09725123-78 2014-08-22 19:25:12,697.697 INFO:tasks.rados.rados.0.mira097.stdout:674: finishing write tid 4 to mira09725123-78 2014-08-22 19:25:12,697.697 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 78 v 835 (ObjNum 566 snap 20 seq_num 566) dirty exists 2014-08-22 19:25:12,697.697 INFO:tasks.rados.rados.0.mira097.stdout:674: done (0 left) 2014-08-22 19:25:12,932.932 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.4. 2014-08-22 19:25:12,937.937 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:05.846719 7fa1e1182700 20 client.4163.objecter shutdown clearing up homeless session... 2014-08-22 19:25:12,937.937 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:05.847417 7fa1e1182700 1 librados: shutdown 2014-08-22 19:25:13,906.906 INFO:tasks.rados.rados.0.mira097.stdout:678: done (0 left) 2014-08-22 19:25:13,906.906 INFO:tasks.rados.rados.0.mira097.stdout:679: read oid 163 snap -1 2014-08-22 19:25:13,906.906 INFO:tasks.rados.rados.0.mira097.stdout:680: write oid 443 current snap is 21 2014-08-22 19:25:13,906.906 INFO:tasks.rados.rados.0.mira097.stdout:680: seq_num 569 ranges {414076=641630,1814784=400370,2619235=194841} 2014-08-22 19:25:13,939.939 INFO:tasks.rados.rados.0.mira097.stdout:680: writing mira09725123-443 from 414076 to 1055706 tid 1 2014-08-22 19:25:13,956.956 INFO:tasks.rados.rados.0.mira097.stdout:680: writing mira09725123-443 from 1814784 to 2215154 tid 2 2014-08-22 19:25:13,972.972 INFO:tasks.rados.rados.0.mira097.stdout:680: writing mira09725123-443 from 2619235 to 2814076 tid 3 2014-08-22 19:25:13,972.972 INFO:tasks.rados.rados.0.mira097.stdout:681: delete oid 344 current snap is 21 2014-08-22 19:25:13,972.972 INFO:tasks.rados.rados.0.mira097.stdout:679: expect (ObjNum 162 snap 0 seq_num 162) 2014-08-22 19:25:14,981.981 INFO:tasks.rados.rados.0.mira097.stdout:680: finishing write tid 1 to mira09725123-443 2014-08-22 19:25:14,981.981 INFO:tasks.rados.rados.0.mira097.stdout:680: finishing write tid 2 to mira09725123-443 2014-08-22 19:25:14,981.981 INFO:tasks.rados.rados.0.mira097.stdout:680: finishing write tid 3 to mira09725123-443 2014-08-22 19:25:14,981.981 INFO:tasks.rados.rados.0.mira097.stdout:680: finishing write tid 5 to mira09725123-443 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:680: finishing write tid 6 to mira09725123-443 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 443 v 922 (ObjNum 569 snap 21 seq_num 569) dirty exists 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:679: done (2 left) 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:680: done (1 left) 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:681: done (0 left) 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:682: rollback oid 276 current snap is 21 2014-08-22 19:25:14,982.982 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 276 to 13 2014-08-22 19:25:14,983.983 INFO:tasks.rados.rados.0.mira097.stdout:683: snap_create 2014-08-22 19:25:14,983.983 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:17,580.580 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 276 v 842 (ObjNum 275 snap 0 seq_num 275) dirty exists 2014-08-22 19:25:17,581.581 INFO:tasks.rados.rados.0.mira097.stdout:682: done (0 left) 2014-08-22 19:25:17,948.948 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2] out_osds: [0, 4] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:17,948.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:17,949.949 INFO:tasks.thrashosds.thrasher:Adding osd 0 2014-08-22 19:25:17,949.949 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 0' 2014-08-22 19:25:18,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.327080 7fd8e2398700 1 librados: starting msgr at :/0 2014-08-22 19:25:18,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.327176 7fd8e2398700 1 librados: starting objecter 2014-08-22 19:25:18,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.327442 7fd8e2398700 1 librados: setting wanted keys 2014-08-22 19:25:18,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.327449 7fd8e2398700 1 librados: calling monclient init 2014-08-22 19:25:18,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.329289 7fd8daffd700 10 client.?.objecter ms_handle_connect 0x7fd8dc0355e0 2014-08-22 19:25:18,421.421 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.329327 7fd8daffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:18,424.424 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.333425 7fd8e2398700 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:18,425.425 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.333595 7fd8e2398700 1 librados: init done 2014-08-22 19:25:18,425.425 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334431 7fd8daffd700 10 client.4177.objecter ms_dispatch 0x7fd8dc000f80 osd_map(76..76 src has 1..76) v3 2014-08-22 19:25:18,425.425 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334450 7fd8daffd700 3 client.4177.objecter handle_osd_map got epochs [76,76] > 0 2014-08-22 19:25:18,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334455 7fd8daffd700 3 client.4177.objecter handle_osd_map decoding full epoch 76 2014-08-22 19:25:18,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334631 7fd8daffd700 20 client.4177.objecter dump_active .. 0 homeless 2014-08-22 19:25:18,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334717 7fd8daffd700 10 client.4177.objecter ms_dispatch 0x7fd8dc000f80 osd_map(76..76 src has 1..76) v3 2014-08-22 19:25:18,498.498 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334730 7fd8daffd700 3 client.4177.objecter handle_osd_map ignoring epochs [76,76] <= 76 2014-08-22 19:25:18,498.498 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.334735 7fd8daffd700 20 client.4177.objecter dump_active .. 0 homeless 2014-08-22 19:25:18,755.755 INFO:tasks.rados.rados.0.mira097.stdout:683: done (0 left) 2014-08-22 19:25:18,755.755 INFO:tasks.rados.rados.0.mira097.stdout:684: rollback oid 458 current snap is 22 2014-08-22 19:25:18,755.755 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 458 to 14 2014-08-22 19:25:18,755.755 INFO:tasks.rados.rados.0.mira097.stdout:685: delete oid 466 current snap is 22 2014-08-22 19:25:18,777.777 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 458 v 846 (ObjNum 457 snap 0 seq_num 457) dirty exists 2014-08-22 19:25:18,777.777 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.0. 2014-08-22 19:25:18,781.781 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.690516 7fd8e2398700 20 client.4177.objecter shutdown clearing up homeless session... 2014-08-22 19:25:18,781.781 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:11.690906 7fd8e2398700 1 librados: shutdown 2014-08-22 19:25:18,787.787 INFO:tasks.thrashosds.thrasher:Added osd 0 2014-08-22 19:25:19,513.513 INFO:tasks.rados.rados.0.mira097.stdout:684: done (1 left) 2014-08-22 19:25:19,514.514 INFO:tasks.rados.rados.0.mira097.stdout:685: done (0 left) 2014-08-22 19:25:19,514.514 INFO:tasks.rados.rados.0.mira097.stdout:686: rollback oid 268 current snap is 22 2014-08-22 19:25:19,514.514 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 268 to 16 2014-08-22 19:25:19,514.514 INFO:tasks.rados.rados.0.mira097.stdout:687: rollback oid 462 current snap is 22 2014-08-22 19:25:19,514.514 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 462 to 21 2014-08-22 19:25:19,515.515 INFO:tasks.rados.rados.0.mira097.stdout:688: delete oid 416 current snap is 22 2014-08-22 19:25:19,561.561 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 462 v 924 (ObjNum 461 snap 0 seq_num 461) dirty exists 2014-08-22 19:25:19,922.922 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 268 v 1021 (ObjNum 267 snap 0 seq_num 267) dirty exists 2014-08-22 19:25:20,030.030 INFO:tasks.rados.rados.0.mira097.stdout:686: done (2 left) 2014-08-22 19:25:20,030.030 INFO:tasks.rados.rados.0.mira097.stdout:687: done (1 left) 2014-08-22 19:25:20,031.031 INFO:tasks.rados.rados.0.mira097.stdout:688: done (0 left) 2014-08-22 19:25:20,031.031 INFO:tasks.rados.rados.0.mira097.stdout:689: snap_create 2014-08-22 19:25:21,085.085 INFO:tasks.rados.rados.0.mira097.stdout:689: done (0 left) 2014-08-22 19:25:21,085.085 INFO:tasks.rados.rados.0.mira097.stdout:690: write oid 155 current snap is 23 2014-08-22 19:25:21,085.085 INFO:tasks.rados.rados.0.mira097.stdout:690: seq_num 573 ranges {209522=1} 2014-08-22 19:25:21,085.085 INFO:tasks.rados.rados.0.mira097.stdout:690: writing mira09725123-155 from 209522 to 209523 tid 1 2014-08-22 19:25:21,086.086 INFO:tasks.rados.rados.0.mira097.stdout:691: write oid 429 current snap is 23 2014-08-22 19:25:21,086.086 INFO:tasks.rados.rados.0.mira097.stdout:691: seq_num 574 ranges {554353=400000} 2014-08-22 19:25:21,100.100 INFO:tasks.rados.rados.0.mira097.stdout:691: writing mira09725123-429 from 554353 to 954353 tid 1 2014-08-22 19:25:21,101.101 INFO:tasks.rados.rados.0.mira097.stdout:692: copy_from oid 144 from oid 231 current snap is 23 2014-08-22 19:25:21,102.102 INFO:tasks.rados.rados.0.mira097.stdout:693: read oid 75 snap -1 2014-08-22 19:25:21,102.102 INFO:tasks.rados.rados.0.mira097.stdout:694: write oid 132 current snap is 23 2014-08-22 19:25:21,102.102 INFO:tasks.rados.rados.0.mira097.stdout:694: seq_num 575 ranges {498159=695048,1698158=1} 2014-08-22 19:25:21,126.126 INFO:tasks.rados.rados.0.mira097.stdout:694: writing mira09725123-132 from 498159 to 1193207 tid 1 2014-08-22 19:25:21,130.130 INFO:tasks.rados.rados.0.mira097.stdout:694: writing mira09725123-132 from 1698158 to 1698159 tid 2 2014-08-22 19:25:21,130.130 INFO:tasks.rados.rados.0.mira097.stdout:695: read oid 24 snap -1 2014-08-22 19:25:21,131.131 INFO:tasks.rados.rados.0.mira097.stdout:696: read oid 103 snap -1 2014-08-22 19:25:21,131.131 INFO:tasks.rados.rados.0.mira097.stdout:697: write oid 12 current snap is 23 2014-08-22 19:25:21,131.131 INFO:tasks.rados.rados.0.mira097.stdout:697: seq_num 576 ranges {594412=667877,1394411=1} 2014-08-22 19:25:21,158.158 INFO:tasks.rados.rados.0.mira097.stdout:697: writing mira09725123-12 from 594412 to 1262289 tid 1 2014-08-22 19:25:21,261.261 INFO:tasks.rados.rados.0.mira097.stdout:697: writing mira09725123-12 from 1394411 to 1394412 tid 2 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:698: read oid 449 snap -1 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:691: finishing write tid 1 to mira09725123-429 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:691: finishing write tid 3 to mira09725123-429 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:691: finishing write tid 4 to mira09725123-429 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 429 v 927 (ObjNum 574 snap 23 seq_num 574) dirty exists 2014-08-22 19:25:21,262.262 INFO:tasks.rados.rados.0.mira097.stdout:691: done (8 left) 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:699: copy_from oid 18 from oid 93 current snap is 23 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:700: read oid 414 snap 13 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:701: write oid 147 current snap is 23 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:701: seq_num 577 ranges {690666=400000} 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:701: writing mira09725123-147 from 690666 to 1090666 tid 1 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:702: write oid 139 current snap is 23 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:702: seq_num 578 ranges {599400=611485,1878389=569023,3154744=244656} 2014-08-22 19:25:21,263.263 INFO:tasks.rados.rados.0.mira097.stdout:702: writing mira09725123-139 from 599400 to 1210885 tid 1 2014-08-22 19:25:21,264.264 INFO:tasks.rados.rados.0.mira097.stdout:702: writing mira09725123-139 from 1878389 to 2447412 tid 2 2014-08-22 19:25:21,264.264 INFO:tasks.rados.rados.0.mira097.stdout:702: writing mira09725123-139 from 3154744 to 3399400 tid 3 2014-08-22 19:25:21,264.264 INFO:tasks.rados.rados.0.mira097.stdout:703: read oid 411 snap -1 2014-08-22 19:25:21,264.264 INFO:tasks.rados.rados.0.mira097.stdout:704: snap_remove snap 15 2014-08-22 19:25:21,302.302 INFO:tasks.rados.rados.0.mira097.stdout:698: expect (ObjNum 448 snap 0 seq_num 448) 2014-08-22 19:25:22,429.429 INFO:tasks.rados.rados.0.mira097.stdout:690: finishing write tid 1 to mira09725123-155 2014-08-22 19:25:22,429.429 INFO:tasks.rados.rados.0.mira097.stdout:690: finishing write tid 3 to mira09725123-155 2014-08-22 19:25:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:690: finishing write tid 4 to mira09725123-155 2014-08-22 19:25:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 155 v 936 (ObjNum 573 snap 23 seq_num 573) dirty exists 2014-08-22 19:25:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:701: finishing write tid 1 to mira09725123-147 2014-08-22 19:25:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:701: finishing write tid 3 to mira09725123-147 2014-08-22 19:25:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:701: finishing write tid 4 to mira09725123-147 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 147 v 939 (ObjNum 577 snap 23 seq_num 577) dirty exists 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:694: finishing write tid 1 to mira09725123-132 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:694: finishing write tid 2 to mira09725123-132 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:694: finishing write tid 4 to mira09725123-132 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:694: finishing write tid 5 to mira09725123-132 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 132 v 1026 (ObjNum 575 snap 23 seq_num 575) dirty exists 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:702: finishing write tid 1 to mira09725123-139 2014-08-22 19:25:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:693: expect (ObjNum 445 snap 0 seq_num 445) 2014-08-22 19:25:22,601.601 INFO:tasks.rados.rados.0.mira097.stdout:697: finishing write tid 1 to mira09725123-12 2014-08-22 19:25:22,601.601 INFO:tasks.rados.rados.0.mira097.stdout:702: finishing write tid 2 to mira09725123-139 2014-08-22 19:25:22,601.601 INFO:tasks.rados.rados.0.mira097.stdout:697: finishing write tid 2 to mira09725123-12 2014-08-22 19:25:22,602.602 INFO:tasks.rados.rados.0.mira097.stdout:697: finishing write tid 4 to mira09725123-12 2014-08-22 19:25:22,602.602 INFO:tasks.rados.rados.0.mira097.stdout:696: expect (ObjNum 102 snap 0 seq_num 102) 2014-08-22 19:25:22,992.992 INFO:tasks.rados.rados.0.mira097.stdout:697: finishing write tid 5 to mira09725123-12 2014-08-22 19:25:22,992.992 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 12 v 1031 (ObjNum 576 snap 23 seq_num 576) dirty exists 2014-08-22 19:25:22,992.992 INFO:tasks.rados.rados.0.mira097.stdout:702: finishing write tid 3 to mira09725123-139 2014-08-22 19:25:22,992.992 INFO:tasks.rados.rados.0.mira097.stdout:702: finishing write tid 5 to mira09725123-139 2014-08-22 19:25:22,993.993 INFO:tasks.rados.rados.0.mira097.stdout:702: finishing write tid 6 to mira09725123-139 2014-08-22 19:25:22,993.993 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 139 v 946 (ObjNum 578 snap 23 seq_num 578) dirty exists 2014-08-22 19:25:22,993.993 INFO:tasks.rados.rados.0.mira097.stdout:700: expect (ObjNum 413 snap 0 seq_num 413) 2014-08-22 19:25:23,787.787 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0] out_osds: [4] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:23,788.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:23,788.788 INFO:tasks.thrashosds.thrasher:Adding osd 4 2014-08-22 19:25:23,788.788 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 4' 2014-08-22 19:25:24,606.606 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.514688 7fc83b844700 1 librados: starting msgr at :/0 2014-08-22 19:25:24,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.514695 7fc83b844700 1 librados: starting objecter 2014-08-22 19:25:24,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.514816 7fc83b844700 1 librados: setting wanted keys 2014-08-22 19:25:24,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.514823 7fc83b844700 1 librados: calling monclient init 2014-08-22 19:25:24,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.515738 7fc8388f1700 10 client.?.objecter ms_handle_connect 0x7fc8340355e0 2014-08-22 19:25:24,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.515758 7fc8388f1700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:24,655.655 INFO:tasks.rados.rados.0.mira097.stdout:699: finishing copy_from to mira09725123-18 2014-08-22 19:25:24,655.655 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 18 v 93 (ObjNum 92 snap 0 seq_num 92) dirty exists 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:690: done (13 left) 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:693: done (12 left) 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:694: done (11 left) 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:696: done (10 left) 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:697: done (9 left) 2014-08-22 19:25:24,685.685 INFO:tasks.rados.rados.0.mira097.stdout:698: done (8 left) 2014-08-22 19:25:24,686.686 INFO:tasks.rados.rados.0.mira097.stdout:700: done (7 left) 2014-08-22 19:25:24,686.686 INFO:tasks.rados.rados.0.mira097.stdout:701: done (6 left) 2014-08-22 19:25:24,686.686 INFO:tasks.rados.rados.0.mira097.stdout:702: done (5 left) 2014-08-22 19:25:24,686.686 INFO:tasks.rados.rados.0.mira097.stdout:703: done (4 left) 2014-08-22 19:25:24,687.687 INFO:tasks.rados.rados.0.mira097.stdout:704: done (3 left) 2014-08-22 19:25:24,687.687 INFO:tasks.rados.rados.0.mira097.stdout:705: write oid 286 current snap is 23 2014-08-22 19:25:24,687.687 INFO:tasks.rados.rados.0.mira097.stdout:705: seq_num 579 ranges {695653=618362,1841097=639218,3089217=6436} 2014-08-22 19:25:24,709.709 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.618201 7fc83b844700 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.618359 7fc83b844700 1 librados: init done 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621356 7fc8388f1700 10 client.4166.objecter ms_dispatch 0x7fc834000f80 osd_map(80..80 src has 1..80) v3 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621368 7fc8388f1700 3 client.4166.objecter handle_osd_map got epochs [80,80] > 0 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621372 7fc8388f1700 3 client.4166.objecter handle_osd_map decoding full epoch 80 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621491 7fc8388f1700 20 client.4166.objecter dump_active .. 0 homeless 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621559 7fc8388f1700 10 client.4166.objecter ms_dispatch 0x7fc834000f80 osd_map(80..80 src has 1..80) v3 2014-08-22 19:25:24,780.780 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621561 7fc8388f1700 3 client.4166.objecter handle_osd_map ignoring epochs [80,80] <= 80 2014-08-22 19:25:24,782.782 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:17.621563 7fc8388f1700 20 client.4166.objecter dump_active .. 0 homeless 2014-08-22 19:25:24,782.782 INFO:tasks.rados.rados.0.mira097.stdout:705: writing mira09725123-286 from 695653 to 1314015 tid 1 2014-08-22 19:25:24,782.782 INFO:tasks.rados.rados.0.mira097.stdout:705: writing mira09725123-286 from 1841097 to 2480315 tid 2 2014-08-22 19:25:24,782.782 INFO:tasks.rados.rados.0.mira097.stdout:705: writing mira09725123-286 from 3089217 to 3095653 tid 3 2014-08-22 19:25:24,782.782 INFO:tasks.rados.rados.0.mira097.stdout:706: snap_remove snap 21 2014-08-22 19:25:24,782.782 INFO:tasks.rados.rados.0.mira097.stdout:699: finishing copy_from racing read to mira09725123-18 2014-08-22 19:25:24,783.783 INFO:tasks.rados.rados.0.mira097.stdout:695: expect (ObjNum 23 snap 0 seq_num 23) 2014-08-22 19:25:25,787.787 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.4. 2014-08-22 19:25:25,788.788 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:18.697254 7fc83b844700 20 client.4166.objecter shutdown clearing up homeless session... 2014-08-22 19:25:25,788.788 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:18.697776 7fc83b844700 1 librados: shutdown 2014-08-22 19:25:25,796.796 INFO:tasks.thrashosds.thrasher:Added osd 4 2014-08-22 19:25:26,004.004 INFO:tasks.rados.rados.0.mira097.stdout:692: finishing copy_from to mira09725123-144 2014-08-22 19:25:26,004.004 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 144 v 245 (ObjNum 230 snap 0 seq_num 230) dirty exists 2014-08-22 19:25:26,005.005 INFO:tasks.rados.rados.0.mira097.stdout:692: finishing copy_from racing read to mira09725123-144 2014-08-22 19:25:26,005.005 INFO:tasks.rados.rados.0.mira097.stdout:705: finishing write tid 1 to mira09725123-286 2014-08-22 19:25:26,005.005 INFO:tasks.rados.rados.0.mira097.stdout:692: done (4 left) 2014-08-22 19:25:26,005.005 INFO:tasks.rados.rados.0.mira097.stdout:695: done (3 left) 2014-08-22 19:25:26,006.006 INFO:tasks.rados.rados.0.mira097.stdout:699: done (2 left) 2014-08-22 19:25:26,006.006 INFO:tasks.rados.rados.0.mira097.stdout:706: done (1 left) 2014-08-22 19:25:26,006.006 INFO:tasks.rados.rados.0.mira097.stdout:707: copy_from oid 375 from oid 424 current snap is 23 2014-08-22 19:25:26,006.006 INFO:tasks.rados.rados.0.mira097.stdout:705: finishing write tid 2 to mira09725123-286 2014-08-22 19:25:26,006.006 INFO:tasks.rados.rados.0.mira097.stdout:705: finishing write tid 3 to mira09725123-286 2014-08-22 19:25:26,007.007 INFO:tasks.rados.rados.0.mira097.stdout:705: finishing write tid 5 to mira09725123-286 2014-08-22 19:25:26,007.007 INFO:tasks.rados.rados.0.mira097.stdout:705: finishing write tid 6 to mira09725123-286 2014-08-22 19:25:26,007.007 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 286 v 975 (ObjNum 579 snap 23 seq_num 579) dirty exists 2014-08-22 19:25:26,007.007 INFO:tasks.rados.rados.0.mira097.stdout:705: done (1 left) 2014-08-22 19:25:26,008.008 INFO:tasks.rados.rados.0.mira097.stdout:708: rollback oid 484 current snap is 23 2014-08-22 19:25:26,008.008 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 484 to 14 2014-08-22 19:25:26,008.008 INFO:tasks.rados.rados.0.mira097.stdout:709: write oid 143 current snap is 23 2014-08-22 19:25:26,008.008 INFO:tasks.rados.rados.0.mira097.stdout:709: seq_num 580 ranges {790883=742614,2106653=460837,2790882=1} 2014-08-22 19:25:26,034.034 INFO:tasks.rados.rados.0.mira097.stdout:709: writing mira09725123-143 from 790883 to 1533497 tid 1 2014-08-22 19:25:26,052.052 INFO:tasks.rados.rados.0.mira097.stdout:709: writing mira09725123-143 from 2106653 to 2567490 tid 2 2014-08-22 19:25:26,054.054 INFO:tasks.rados.rados.0.mira097.stdout:709: writing mira09725123-143 from 2790882 to 2790883 tid 3 2014-08-22 19:25:26,054.054 INFO:tasks.rados.rados.0.mira097.stdout:710: read oid 434 snap -1 2014-08-22 19:25:26,178.178 INFO:tasks.rados.rados.0.mira097.stdout:711: rollback oid 303 current snap is 23 2014-08-22 19:25:26,178.178 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 303 to 19 2014-08-22 19:25:26,178.178 INFO:tasks.rados.rados.0.mira097.stdout:712: read oid 298 snap -1 2014-08-22 19:25:26,178.178 INFO:tasks.rados.rados.0.mira097.stdout:713: read oid 192 snap 20 2014-08-22 19:25:26,178.178 INFO:tasks.rados.rados.0.mira097.stdout:714: snap_create 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 7 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:709: finishing write tid 1 to mira09725123-143 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:709: finishing write tid 2 to mira09725123-143 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:709: finishing write tid 3 to mira09725123-143 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:709: finishing write tid 5 to mira09725123-143 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:709: finishing write tid 6 to mira09725123-143 2014-08-22 19:25:26,179.179 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 143 v 1021 (ObjNum 580 snap 23 seq_num 580) dirty exists 2014-08-22 19:25:26,180.180 INFO:tasks.rados.rados.0.mira097.stdout:709: done (6 left) 2014-08-22 19:25:26,180.180 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:25:26,237.237 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 303 v 878 (ObjNum 302 snap 0 seq_num 302) dirty exists 2014-08-22 19:25:26,237.237 INFO:tasks.rados.rados.0.mira097.stdout:711: done (5 left) 2014-08-22 19:25:26,237.237 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:25:26,276.276 INFO:tasks.rados.rados.0.mira097.stdout:710: expect (ObjNum 433 snap 0 seq_num 433) 2014-08-22 19:25:27,536.536 INFO:tasks.rados.rados.0.mira097.stdout:712: expect (ObjNum 297 snap 0 seq_num 297) 2014-08-22 19:25:27,890.890 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 484 v 1061 (ObjNum 483 snap 0 seq_num 483) dirty exists 2014-08-22 19:25:27,890.890 INFO:tasks.rados.rados.0.mira097.stdout:713: expect (ObjNum 191 snap 0 seq_num 191) 2014-08-22 19:25:28,200.200 INFO:tasks.rados.rados.0.mira097.stdout:708: done (4 left) 2014-08-22 19:25:28,200.200 INFO:tasks.rados.rados.0.mira097.stdout:710: done (3 left) 2014-08-22 19:25:28,200.200 INFO:tasks.rados.rados.0.mira097.stdout:712: done (2 left) 2014-08-22 19:25:28,200.200 INFO:tasks.rados.rados.0.mira097.stdout:713: done (1 left) 2014-08-22 19:25:28,201.201 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:30,492.492 INFO:tasks.rados.rados.0.mira097.stdout:707: finishing copy_from to mira09725123-375 2014-08-22 19:25:30,493.493 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 375 v 515 (ObjNum 423 snap 0 seq_num 423) dirty exists 2014-08-22 19:25:30,493.493 INFO:tasks.rados.rados.0.mira097.stdout:707: finishing copy_from racing read to mira09725123-375 2014-08-22 19:25:30,493.493 INFO:tasks.rados.rados.0.mira097.stdout:707: done (0 left) 2014-08-22 19:25:30,795.795 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:30,795.795 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:30,795.795 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:25:30,796.796 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:25:31,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.172525 7f630db64700 1 librados: starting msgr at :/0 2014-08-22 19:25:31,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.172542 7f630db64700 1 librados: starting objecter 2014-08-22 19:25:31,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.172824 7f630db64700 1 librados: setting wanted keys 2014-08-22 19:25:31,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.172843 7f630db64700 1 librados: calling monclient init 2014-08-22 19:25:31,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.173751 7f63067fc700 10 client.?.objecter ms_handle_connect 0x7f63080355e0 2014-08-22 19:25:31,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.173774 7f63067fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:31,267.267 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176201 7f630db64700 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:31,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176310 7f630db64700 1 librados: init done 2014-08-22 19:25:31,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176614 7f63067fc700 10 client.4169.objecter ms_dispatch 0x7f6308000f80 osd_map(82..82 src has 1..82) v3 2014-08-22 19:25:31,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176620 7f63067fc700 3 client.4169.objecter handle_osd_map got epochs [82,82] > 0 2014-08-22 19:25:31,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176621 7f63067fc700 3 client.4169.objecter handle_osd_map decoding full epoch 82 2014-08-22 19:25:31,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176690 7f63067fc700 20 client.4169.objecter dump_active .. 0 homeless 2014-08-22 19:25:31,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176795 7f63067fc700 10 client.4169.objecter ms_dispatch 0x7f6308000f80 osd_map(82..82 src has 1..82) v3 2014-08-22 19:25:31,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176799 7f63067fc700 3 client.4169.objecter handle_osd_map ignoring epochs [82,82] <= 82 2014-08-22 19:25:31,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.176800 7f63067fc700 20 client.4169.objecter dump_active .. 0 homeless 2014-08-22 19:25:31,464.464 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:25:31,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.264583 7f630db64700 20 client.4169.objecter shutdown clearing up homeless session... 2014-08-22 19:25:31,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:24.265468 7f630db64700 1 librados: shutdown 2014-08-22 19:25:31,815.815 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:25:31,816.816 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 54' 2014-08-22 19:25:32,274.274 INFO:tasks.rados.rados.0.mira097.stdout:714: done (0 left) 2014-08-22 19:25:32,275.275 INFO:tasks.rados.rados.0.mira097.stdout:715: rollback oid 295 current snap is 24 2014-08-22 19:25:32,275.275 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 295 to 20 2014-08-22 19:25:32,275.275 INFO:tasks.rados.rados.0.mira097.stdout:716: snap_remove snap 13 2014-08-22 19:25:32,318.318 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 295 v 993 (ObjNum 294 snap 0 seq_num 294) dirty exists 2014-08-22 19:25:32,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.331510 7f5b4c4c4700 1 librados: starting msgr at :/0 2014-08-22 19:25:32,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.331523 7f5b4c4c4700 1 librados: starting objecter 2014-08-22 19:25:32,424.424 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.331701 7f5b4c4c4700 1 librados: setting wanted keys 2014-08-22 19:25:32,424.424 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.331705 7f5b4c4c4700 1 librados: calling monclient init 2014-08-22 19:25:32,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.334800 7f5b49571700 10 client.?.objecter ms_handle_connect 0x7f5b440355e0 2014-08-22 19:25:32,427.427 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.334847 7f5b49571700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:32,427.427 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.336487 7f5b4c4c4700 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:32,428.428 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.336769 7f5b4c4c4700 1 librados: init done 2014-08-22 19:25:32,429.429 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.337999 7f5b49571700 10 client.4170.objecter ms_dispatch 0x7f5b44000f80 osd_map(83..83 src has 1..83) v3 2014-08-22 19:25:32,430.430 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338075 7f5b49571700 3 client.4170.objecter handle_osd_map got epochs [83,83] > 0 2014-08-22 19:25:32,430.430 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338084 7f5b49571700 3 client.4170.objecter handle_osd_map decoding full epoch 83 2014-08-22 19:25:32,430.430 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338233 7f5b49571700 20 client.4170.objecter dump_active .. 0 homeless 2014-08-22 19:25:32,430.430 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338292 7f5b49571700 10 client.4170.objecter ms_dispatch 0x7f5b44000f80 osd_map(83..83 src has 1..83) v3 2014-08-22 19:25:32,430.430 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338301 7f5b49571700 3 client.4170.objecter handle_osd_map ignoring epochs [83,83] <= 83 2014-08-22 19:25:32,431.431 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:25.338304 7f5b49571700 20 client.4170.objecter dump_active .. 0 homeless 2014-08-22 19:25:37,368.368 INFO:tasks.rados.rados.0.mira097.stdout:715: done (1 left) 2014-08-22 19:25:37,368.368 INFO:tasks.rados.rados.0.mira097.stdout:716: done (0 left) 2014-08-22 19:25:37,368.368 INFO:tasks.rados.rados.0.mira097.stdout:717: snap_create 2014-08-22 19:25:37,388.388 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 54 2014-08-22 19:25:37,389.389 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:30.297499 7f5b4c4c4700 20 client.4170.objecter shutdown clearing up homeless session... 2014-08-22 19:25:37,390.390 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:30.298599 7f5b4c4c4700 1 librados: shutdown 2014-08-22 19:25:41,822.822 INFO:tasks.rados.rados.0.mira097.stdout:717: done (0 left) 2014-08-22 19:25:41,823.823 INFO:tasks.rados.rados.0.mira097.stdout:718: delete oid 311 current snap is 25 2014-08-22 19:25:41,837.837 INFO:tasks.rados.rados.0.mira097.stdout:718: done (0 left) 2014-08-22 19:25:41,837.837 INFO:tasks.rados.rados.0.mira097.stdout:719: delete oid 189 current snap is 25 2014-08-22 19:25:41,871.871 INFO:tasks.rados.rados.0.mira097.stdout:719: done (0 left) 2014-08-22 19:25:41,871.871 INFO:tasks.rados.rados.0.mira097.stdout:720: snap_remove snap 16 2014-08-22 19:25:42,400.400 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:42,401.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:42,401.401 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-22 19:25:42,401.401 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:25:42,402.402 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:25:42,402.402 INFO:teuthology.orchestra.run.mira097:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-22 19:25:42,866.866 INFO:tasks.rados.rados.0.mira097.stdout:720: done (0 left) 2014-08-22 19:25:42,866.866 INFO:tasks.rados.rados.0.mira097.stdout:721: read oid 14 snap -1 2014-08-22 19:25:42,867.867 INFO:tasks.rados.rados.0.mira097.stdout:722: copy_from oid 180 from oid 127 current snap is 25 2014-08-22 19:25:42,867.867 INFO:tasks.rados.rados.0.mira097.stdout:723: read oid 229 snap -1 2014-08-22 19:25:42,867.867 INFO:tasks.rados.rados.0.mira097.stdout:724: rollback oid 441 current snap is 25 2014-08-22 19:25:42,867.867 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 441 to 22 2014-08-22 19:25:42,873.873 INFO:tasks.rados.rados.0.mira097.stdout:725: write oid 263 current snap is 25 2014-08-22 19:25:42,874.874 INFO:tasks.rados.rados.0.mira097.stdout:725: seq_num 583 ranges {491099=1} 2014-08-22 19:25:42,874.874 INFO:tasks.rados.rados.0.mira097.stdout:725: writing mira09725123-263 from 491099 to 491100 tid 1 2014-08-22 19:25:42,874.874 INFO:tasks.rados.rados.0.mira097.stdout:726: write oid 85 current snap is 25 2014-08-22 19:25:42,874.874 INFO:tasks.rados.rados.0.mira097.stdout:726: seq_num 584 ranges {684506=417351,1539566=493042,2284505=1} 2014-08-22 19:25:42,893.893 INFO:tasks.rados.rados.0.mira097.stdout:726: writing mira09725123-85 from 684506 to 1101857 tid 1 2014-08-22 19:25:42,911.911 INFO:tasks.rados.rados.0.mira097.stdout:726: writing mira09725123-85 from 1539566 to 2032608 tid 2 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:726: writing mira09725123-85 from 2284505 to 2284506 tid 3 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:727: rollback oid 8 current snap is 25 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 8 to 23 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 441 v 1040 (ObjNum 440 snap 0 seq_num 440) dirty exists 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:724: done (6 left) 2014-08-22 19:25:42,957.957 INFO:tasks.rados.rados.0.mira097.stdout:728: read oid 449 snap 20 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:729: write oid 19 current snap is 25 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:729: seq_num 585 ranges {779736=789156,1979735=1} 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:729: writing mira09725123-19 from 779736 to 1568892 tid 1 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:729: writing mira09725123-19 from 1979735 to 1979736 tid 2 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:730: read oid 454 snap -1 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:731: rollback oid 90 current snap is 25 2014-08-22 19:25:42,958.958 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 90 to 14 2014-08-22 19:25:42,959.959 INFO:tasks.rados.rados.0.mira097.stdout:732: write oid 388 current snap is 25 2014-08-22 19:25:42,959.959 INFO:tasks.rados.rados.0.mira097.stdout:732: seq_num 586 ranges {475990=512384,1675989=1} 2014-08-22 19:25:42,966.966 INFO:tasks.rados.rados.0.mira097.stdout:732: writing mira09725123-388 from 475990 to 988374 tid 1 2014-08-22 19:25:43,039.039 INFO:tasks.rados.rados.0.mira097.stdout:732: writing mira09725123-388 from 1675989 to 1675990 tid 2 2014-08-22 19:25:43,039.039 INFO:tasks.rados.rados.0.mira097.stdout:733: read oid 22 snap -1 2014-08-22 19:25:43,039.039 INFO:tasks.rados.rados.0.mira097.stdout:734: read oid 310 snap 20 2014-08-22 19:25:43,039.039 INFO:tasks.rados.rados.0.mira097.stdout:735: snap_remove snap 24 2014-08-22 19:25:43,092.092 INFO:tasks.rados.rados.0.mira097.stdout:728: expect (ObjNum 448 snap 0 seq_num 448) 2014-08-22 19:25:44,223.223 INFO:tasks.rados.rados.0.mira097.stdout:721: expect (ObjNum 13 snap 0 seq_num 13) 2014-08-22 19:25:45,233.233 INFO:tasks.rados.rados.0.mira097.stdout:725: finishing write tid 1 to mira09725123-263 2014-08-22 19:25:45,234.234 INFO:tasks.rados.rados.0.mira097.stdout:725: finishing write tid 3 to mira09725123-263 2014-08-22 19:25:45,234.234 INFO:tasks.rados.rados.0.mira097.stdout:725: finishing write tid 4 to mira09725123-263 2014-08-22 19:25:45,234.234 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 263 v 1089 (ObjNum 583 snap 25 seq_num 583) dirty exists 2014-08-22 19:25:45,234.234 INFO:tasks.rados.rados.0.mira097.stdout:730: expect (ObjNum 453 snap 0 seq_num 453) 2014-08-22 19:25:45,392.392 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 8 v 1043 (ObjNum 162 snap 0 seq_num 162) dirty exists 2014-08-22 19:25:45,392.392 INFO:tasks.rados.rados.0.mira097.stdout:729: finishing write tid 1 to mira09725123-19 2014-08-22 19:25:45,393.393 INFO:tasks.rados.rados.0.mira097.stdout:729: finishing write tid 2 to mira09725123-19 2014-08-22 19:25:45,393.393 INFO:tasks.rados.rados.0.mira097.stdout:729: finishing write tid 4 to mira09725123-19 2014-08-22 19:25:45,393.393 INFO:tasks.rados.rados.0.mira097.stdout:729: finishing write tid 5 to mira09725123-19 2014-08-22 19:25:45,393.393 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 19 v 903 (ObjNum 585 snap 25 seq_num 585) dirty exists 2014-08-22 19:25:45,393.393 INFO:tasks.rados.rados.0.mira097.stdout:733: expect (ObjNum 21 snap 0 seq_num 21) 2014-08-22 19:25:45,915.915 INFO:tasks.rados.rados.0.mira097.stdout:721: done (13 left) 2014-08-22 19:25:45,915.915 INFO:tasks.rados.rados.0.mira097.stdout:725: done (12 left) 2014-08-22 19:25:45,915.915 INFO:tasks.rados.rados.0.mira097.stdout:727: done (11 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:728: done (10 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:729: done (9 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:730: done (8 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:733: done (7 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:735: done (6 left) 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout:736: snap_create 2014-08-22 19:25:45,916.916 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:25:45,917.917 INFO:tasks.rados.rados.0.mira097.stdout:723: expect (ObjNum 228 snap 0 seq_num 228) 2014-08-22 19:25:46,966.966 INFO:tasks.rados.rados.0.mira097.stdout:726: finishing write tid 1 to mira09725123-85 2014-08-22 19:25:46,966.966 INFO:tasks.rados.rados.0.mira097.stdout:726: finishing write tid 2 to mira09725123-85 2014-08-22 19:25:46,966.966 INFO:tasks.rados.rados.0.mira097.stdout:726: finishing write tid 3 to mira09725123-85 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:726: finishing write tid 5 to mira09725123-85 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:723: done (5 left) 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:726: finishing write tid 6 to mira09725123-85 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 85 v 1025 (ObjNum 584 snap 25 seq_num 584) dirty exists 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 90 v 1031 (ObjNum 89 snap 0 seq_num 89) dirty exists 2014-08-22 19:25:46,967.967 INFO:tasks.rados.rados.0.mira097.stdout:732: finishing write tid 1 to mira09725123-388 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:732: finishing write tid 2 to mira09725123-388 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:732: finishing write tid 4 to mira09725123-388 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:722: finishing copy_from to mira09725123-180 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 180 v 163 (ObjNum 126 snap 0 seq_num 126) dirty exists 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:722: finishing copy_from racing read to mira09725123-180 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:732: finishing write tid 5 to mira09725123-388 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 388 v 1039 (ObjNum 586 snap 25 seq_num 586) dirty exists 2014-08-22 19:25:46,968.968 INFO:tasks.rados.rados.0.mira097.stdout:734: expect (ObjNum 309 snap 0 seq_num 309) 2014-08-22 19:25:47,885.885 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:25:47,886.886 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:25:47,887.887 INFO:tasks.rados.rados.0.mira097.stdout:722: done (4 left) 2014-08-22 19:25:47,887.887 INFO:tasks.rados.rados.0.mira097.stdout:726: done (3 left) 2014-08-22 19:25:47,887.887 INFO:tasks.rados.rados.0.mira097.stdout:731: done (2 left) 2014-08-22 19:25:47,887.887 INFO:tasks.rados.rados.0.mira097.stdout:732: done (1 left) 2014-08-22 19:25:47,887.887 INFO:tasks.rados.rados.0.mira097.stdout:734: done (0 left) 2014-08-22 19:25:48,361.361 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.268426 7f595bfbf700 1 librados: starting msgr at :/0 2014-08-22 19:25:48,361.361 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.268444 7f595bfbf700 1 librados: starting objecter 2014-08-22 19:25:48,361.361 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.268678 7f595bfbf700 1 librados: setting wanted keys 2014-08-22 19:25:48,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.268693 7f595bfbf700 1 librados: calling monclient init 2014-08-22 19:25:48,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.270507 7f595906c700 10 client.?.objecter ms_handle_connect 0x7f59540355e0 2014-08-22 19:25:48,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.270553 7f595906c700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:48,365.365 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.273693 7f595bfbf700 10 client.4172.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:48,365.365 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.273897 7f595bfbf700 1 librados: init done 2014-08-22 19:25:48,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274501 7f595906c700 10 client.4172.objecter ms_dispatch 0x7f5954000f80 osd_map(87..87 src has 1..87) v3 2014-08-22 19:25:48,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274527 7f595906c700 3 client.4172.objecter handle_osd_map got epochs [87,87] > 0 2014-08-22 19:25:48,366.366 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274536 7f595906c700 3 client.4172.objecter handle_osd_map decoding full epoch 87 2014-08-22 19:25:48,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274697 7f595906c700 20 client.4172.objecter dump_active .. 0 homeless 2014-08-22 19:25:48,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274757 7f595906c700 10 client.4172.objecter ms_dispatch 0x7f5954000f80 osd_map(87..87 src has 1..87) v3 2014-08-22 19:25:48,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274762 7f595906c700 3 client.4172.objecter handle_osd_map ignoring epochs [87,87] <= 87 2014-08-22 19:25:48,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:41.274765 7f595906c700 20 client.4172.objecter dump_active .. 0 homeless 2014-08-22 19:25:51,168.168 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:25:51,172.172 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.080818 7f595bfbf700 20 client.4172.objecter shutdown clearing up homeless session... 2014-08-22 19:25:51,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.081300 7f595bfbf700 1 librados: shutdown 2014-08-22 19:25:51,176.176 INFO:tasks.rados.rados.0.mira097.stdout:736: done (0 left) 2014-08-22 19:25:51,176.176 INFO:tasks.rados.rados.0.mira097.stdout:737: delete oid 26 current snap is 26 2014-08-22 19:25:51,182.182 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:25:51,299.299 INFO:tasks.rados.rados.0.mira097.stdout:737: done (0 left) 2014-08-22 19:25:51,299.299 INFO:tasks.rados.rados.0.mira097.stdout:738: read oid 254 snap -1 2014-08-22 19:25:51,299.299 INFO:tasks.rados.rados.0.mira097.stdout:739: snap_create 2014-08-22 19:25:51,299.299 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:51,313.313 INFO:tasks.rados.rados.0.mira097.stdout:738: expect (ObjNum 253 snap 0 seq_num 253) 2014-08-22 19:25:51,731.731 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.638952 7fcde5bd7700 1 librados: starting msgr at :/0 2014-08-22 19:25:51,731.731 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.638963 7fcde5bd7700 1 librados: starting objecter 2014-08-22 19:25:51,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.639092 7fcde5bd7700 1 librados: setting wanted keys 2014-08-22 19:25:51,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.639098 7fcde5bd7700 1 librados: calling monclient init 2014-08-22 19:25:51,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.640761 7fcdde7fc700 10 client.?.objecter ms_handle_connect 0x7fcde00355e0 2014-08-22 19:25:51,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.640789 7fcdde7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:51,736.736 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.643975 7fcde5bd7700 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:51,736.736 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.644071 7fcde5bd7700 1 librados: init done 2014-08-22 19:25:51,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645099 7fcdde7fc700 10 client.4180.objecter ms_dispatch 0x7fcde0000f80 osd_map(88..88 src has 1..88) v3 2014-08-22 19:25:51,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645108 7fcdde7fc700 3 client.4180.objecter handle_osd_map got epochs [88,88] > 0 2014-08-22 19:25:51,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645112 7fcdde7fc700 3 client.4180.objecter handle_osd_map decoding full epoch 88 2014-08-22 19:25:51,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645184 7fcdde7fc700 20 client.4180.objecter dump_active .. 0 homeless 2014-08-22 19:25:51,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645213 7fcdde7fc700 10 client.4180.objecter ms_dispatch 0x7fcde0000f80 osd_map(88..88 src has 1..88) v3 2014-08-22 19:25:51,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645215 7fcdde7fc700 3 client.4180.objecter handle_osd_map ignoring epochs [88,88] <= 88 2014-08-22 19:25:51,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:44.645216 7fcdde7fc700 20 client.4180.objecter dump_active .. 0 homeless 2014-08-22 19:25:52,179.179 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:25:52,182.182 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.090544 7fcde5bd7700 20 client.4180.objecter shutdown clearing up homeless session... 2014-08-22 19:25:52,182.182 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.090875 7fcde5bd7700 1 librados: shutdown 2014-08-22 19:25:52,187.187 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:25:52,804.804 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.711222 7fe2cf57a700 1 librados: starting msgr at :/0 2014-08-22 19:25:52,804.804 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.711230 7fe2cf57a700 1 librados: starting objecter 2014-08-22 19:25:52,805.805 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.711355 7fe2cf57a700 1 librados: setting wanted keys 2014-08-22 19:25:52,805.805 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.711360 7fe2cf57a700 1 librados: calling monclient init 2014-08-22 19:25:52,805.805 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.712171 7fe2bffff700 10 client.?.objecter ms_handle_connect 0x7fe2c80355e0 2014-08-22 19:25:52,805.805 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.712195 7fe2bffff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:52,806.806 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.713837 7fe2cf57a700 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:52,806.806 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.713927 7fe2cf57a700 1 librados: init done 2014-08-22 19:25:52,806.806 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714207 7fe2bffff700 10 client.4173.objecter ms_dispatch 0x7fe2c8000f80 osd_map(89..89 src has 1..89) v3 2014-08-22 19:25:52,807.807 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714214 7fe2bffff700 3 client.4173.objecter handle_osd_map got epochs [89,89] > 0 2014-08-22 19:25:52,807.807 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714217 7fe2bffff700 3 client.4173.objecter handle_osd_map decoding full epoch 89 2014-08-22 19:25:52,807.807 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714293 7fe2bffff700 20 client.4173.objecter dump_active .. 0 homeless 2014-08-22 19:25:52,807.807 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714356 7fe2bffff700 10 client.4173.objecter ms_dispatch 0x7fe2c8000f80 osd_map(89..89 src has 1..89) v3 2014-08-22 19:25:52,808.808 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714360 7fe2bffff700 3 client.4173.objecter handle_osd_map ignoring epochs [89,89] <= 89 2014-08-22 19:25:52,808.808 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:45.714363 7fe2bffff700 20 client.4173.objecter dump_active .. 0 homeless 2014-08-22 19:25:53,196.196 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:25:53,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.107720 7fe2cf57a700 20 client.4173.objecter shutdown clearing up homeless session... 2014-08-22 19:25:53,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.108322 7fe2cf57a700 1 librados: shutdown 2014-08-22 19:25:53,205.205 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:25:53,258.258 INFO:tasks.rados.rados.0.mira097.stdout:738: done (0 left) 2014-08-22 19:25:53,709.709 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.616090 7fb49963e700 1 librados: starting msgr at :/0 2014-08-22 19:25:53,709.709 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.616097 7fb49963e700 1 librados: starting objecter 2014-08-22 19:25:53,710.710 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.616886 7fb49963e700 1 librados: setting wanted keys 2014-08-22 19:25:53,710.710 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.616897 7fb49963e700 1 librados: calling monclient init 2014-08-22 19:25:53,711.711 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.619135 7fb49258b700 10 client.?.objecter ms_handle_connect 0x7fb4940355e0 2014-08-22 19:25:53,711.711 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.619218 7fb49258b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:53,715.715 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.622980 7fb49963e700 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:53,715.715 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.623130 7fb49963e700 1 librados: init done 2014-08-22 19:25:53,716.716 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.623903 7fb49258b700 10 client.4183.objecter ms_dispatch 0x7fb494000f80 osd_map(90..90 src has 1..90) v3 2014-08-22 19:25:53,716.716 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.623918 7fb49258b700 3 client.4183.objecter handle_osd_map got epochs [90,90] > 0 2014-08-22 19:25:53,716.716 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.623922 7fb49258b700 3 client.4183.objecter handle_osd_map decoding full epoch 90 2014-08-22 19:25:53,716.716 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.624041 7fb49258b700 20 client.4183.objecter dump_active .. 0 homeless 2014-08-22 19:25:53,716.716 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.624092 7fb49258b700 10 client.4183.objecter ms_dispatch 0x7fb494000f80 osd_map(90..90 src has 1..90) v3 2014-08-22 19:25:53,717.717 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.624096 7fb49258b700 3 client.4183.objecter handle_osd_map ignoring epochs [90,90] <= 90 2014-08-22 19:25:53,717.717 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:46.624098 7fb49258b700 20 client.4183.objecter dump_active .. 0 homeless 2014-08-22 19:25:54,208.208 INFO:tasks.rados.rados.0.mira097.stdout:739: done (0 left) 2014-08-22 19:25:54,208.208 INFO:tasks.rados.rados.0.mira097.stdout:740: read oid 276 snap -1 2014-08-22 19:25:54,209.209 INFO:tasks.rados.rados.0.mira097.stdout:741: rollback oid 67 current snap is 27 2014-08-22 19:25:54,209.209 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 67 to 20 2014-08-22 19:25:54,209.209 INFO:tasks.rados.rados.0.mira097.stdout:742: write oid 48 current snap is 27 2014-08-22 19:25:54,210.210 INFO:tasks.rados.rados.0.mira097.stdout:742: seq_num 588 ranges {480977=492089,1408575=560319,2481046=618742,3517343=163634} 2014-08-22 19:25:54,219.219 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:25:54,223.223 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.130633 7fb49963e700 20 client.4183.objecter shutdown clearing up homeless session... 2014-08-22 19:25:54,223.223 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.130983 7fb49963e700 1 librados: shutdown 2014-08-22 19:25:54,228.228 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:25:54,230.230 INFO:tasks.rados.rados.0.mira097.stdout:742: writing mira09725123-48 from 480977 to 973066 tid 1 2014-08-22 19:25:54,251.251 INFO:tasks.rados.rados.0.mira097.stdout:742: writing mira09725123-48 from 1408575 to 1968894 tid 2 2014-08-22 19:25:54,276.276 INFO:tasks.rados.rados.0.mira097.stdout:742: writing mira09725123-48 from 2481046 to 3099788 tid 3 2014-08-22 19:25:54,284.284 INFO:tasks.rados.rados.0.mira097.stdout:742: writing mira09725123-48 from 3517343 to 3680977 tid 4 2014-08-22 19:25:54,383.383 INFO:tasks.rados.rados.0.mira097.stdout:743: write oid 207 current snap is 27 2014-08-22 19:25:54,383.383 INFO:tasks.rados.rados.0.mira097.stdout:743: seq_num 589 ranges {424782=1} 2014-08-22 19:25:54,383.383 INFO:tasks.rados.rados.0.mira097.stdout:743: writing mira09725123-207 from 424782 to 424783 tid 1 2014-08-22 19:25:54,384.384 INFO:tasks.rados.rados.0.mira097.stdout:744: snap_remove snap 19 2014-08-22 19:25:54,384.384 INFO:tasks.rados.rados.0.mira097.stdout:740: expect (ObjNum 275 snap 0 seq_num 275) 2014-08-22 19:25:54,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.582440 7f4fa857e700 1 librados: starting msgr at :/0 2014-08-22 19:25:54,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.582445 7f4fa857e700 1 librados: starting objecter 2014-08-22 19:25:54,676.676 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.582832 7f4fa857e700 1 librados: setting wanted keys 2014-08-22 19:25:54,676.676 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.582835 7f4fa857e700 1 librados: calling monclient init 2014-08-22 19:25:54,676.676 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.584555 7f4fa562b700 10 client.?.objecter ms_handle_connect 0x7f4fa00355e0 2014-08-22 19:25:54,676.676 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.584582 7f4fa562b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:54,681.681 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.587999 7f4fa857e700 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:54,681.681 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.588083 7f4fa857e700 1 librados: init done 2014-08-22 19:25:54,681.681 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.588827 7f4fa562b700 10 client.4186.objecter ms_dispatch 0x7f4fa0000f80 osd_map(91..91 src has 1..91) v3 2014-08-22 19:25:54,681.681 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.588838 7f4fa562b700 3 client.4186.objecter handle_osd_map got epochs [91,91] > 0 2014-08-22 19:25:54,682.682 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.588841 7f4fa562b700 3 client.4186.objecter handle_osd_map decoding full epoch 91 2014-08-22 19:25:54,682.682 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.588943 7f4fa562b700 20 client.4186.objecter dump_active .. 0 homeless 2014-08-22 19:25:54,682.682 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.589066 7f4fa562b700 10 client.4186.objecter ms_dispatch 0x7f4fa0000f80 osd_map(91..91 src has 1..91) v3 2014-08-22 19:25:54,682.682 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.589073 7f4fa562b700 3 client.4186.objecter handle_osd_map ignoring epochs [91,91] <= 91 2014-08-22 19:25:54,682.682 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:47.589076 7f4fa562b700 20 client.4186.objecter dump_active .. 0 homeless 2014-08-22 19:25:55,175.175 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 67 v 1112 (ObjNum 66 snap 0 seq_num 66) dirty exists 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 1 to mira09725123-48 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 2 to mira09725123-48 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 3 to mira09725123-48 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 4 to mira09725123-48 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 6 to mira09725123-48 2014-08-22 19:25:55,176.176 INFO:tasks.rados.rados.0.mira097.stdout:742: finishing write tid 7 to mira09725123-48 2014-08-22 19:25:55,177.177 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 48 v 919 (ObjNum 588 snap 27 seq_num 588) dirty exists 2014-08-22 19:25:55,177.177 INFO:tasks.rados.rados.0.mira097.stdout:743: finishing write tid 1 to mira09725123-207 2014-08-22 19:25:55,177.177 INFO:tasks.rados.rados.0.mira097.stdout:743: finishing write tid 3 to mira09725123-207 2014-08-22 19:25:55,177.177 INFO:tasks.rados.rados.0.mira097.stdout:743: finishing write tid 4 to mira09725123-207 2014-08-22 19:25:55,177.177 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 207 v 923 (ObjNum 589 snap 27 seq_num 589) dirty exists 2014-08-22 19:25:55,213.213 INFO:tasks.rados.rados.0.mira097.stdout:740: done (4 left) 2014-08-22 19:25:55,214.214 INFO:tasks.rados.rados.0.mira097.stdout:741: done (3 left) 2014-08-22 19:25:55,214.214 INFO:tasks.rados.rados.0.mira097.stdout:742: done (2 left) 2014-08-22 19:25:55,215.215 INFO:tasks.rados.rados.0.mira097.stdout:743: done (1 left) 2014-08-22 19:25:55,215.215 INFO:tasks.rados.rados.0.mira097.stdout:744: done (0 left) 2014-08-22 19:25:55,215.215 INFO:tasks.rados.rados.0.mira097.stdout:745: rollback oid 264 current snap is 27 2014-08-22 19:25:55,215.215 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 264 to 14 2014-08-22 19:25:55,215.215 INFO:tasks.rados.rados.0.mira097.stdout:746: snap_remove snap 22 2014-08-22 19:25:55,237.237 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:25:55,240.240 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.148050 7f4fa857e700 20 client.4186.objecter shutdown clearing up homeless session... 2014-08-22 19:25:55,240.240 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.148420 7f4fa857e700 1 librados: shutdown 2014-08-22 19:25:55,246.246 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:25:55,269.269 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 264 v 951 (ObjNum 263 snap 0 seq_num 263) dirty exists 2014-08-22 19:25:55,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.628066 7f6224d87700 1 librados: starting msgr at :/0 2014-08-22 19:25:55,721.721 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.628074 7f6224d87700 1 librados: starting objecter 2014-08-22 19:25:55,721.721 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.628426 7f6224d87700 1 librados: setting wanted keys 2014-08-22 19:25:55,721.721 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.628439 7f6224d87700 1 librados: calling monclient init 2014-08-22 19:25:55,722.722 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.630931 7f621dd8a700 10 client.?.objecter ms_handle_connect 0x7f62200355e0 2014-08-22 19:25:55,722.722 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.630973 7f621dd8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:55,726.726 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.634913 7f6224d87700 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:55,727.727 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.635075 7f6224d87700 1 librados: init done 2014-08-22 19:25:55,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.635880 7f621dd8a700 10 client.4189.objecter ms_dispatch 0x7f6220000f80 osd_map(92..92 src has 1..92) v3 2014-08-22 19:25:55,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.635892 7f621dd8a700 3 client.4189.objecter handle_osd_map got epochs [92,92] > 0 2014-08-22 19:25:55,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.635897 7f621dd8a700 3 client.4189.objecter handle_osd_map decoding full epoch 92 2014-08-22 19:25:55,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.636011 7f621dd8a700 20 client.4189.objecter dump_active .. 0 homeless 2014-08-22 19:25:55,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.636070 7f621dd8a700 10 client.4189.objecter ms_dispatch 0x7f6220000f80 osd_map(92..92 src has 1..92) v3 2014-08-22 19:25:55,729.729 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.636077 7f621dd8a700 3 client.4189.objecter handle_osd_map ignoring epochs [92,92] <= 92 2014-08-22 19:25:55,729.729 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:48.636081 7f621dd8a700 20 client.4189.objecter dump_active .. 0 homeless 2014-08-22 19:25:56,225.225 INFO:tasks.rados.rados.0.mira097.stdout:745: done (1 left) 2014-08-22 19:25:56,226.226 INFO:tasks.rados.rados.0.mira097.stdout:746: done (0 left) 2014-08-22 19:25:56,226.226 INFO:tasks.rados.rados.0.mira097.stdout:747: copy_from oid 246 from oid 78 current snap is 27 2014-08-22 19:25:56,226.226 INFO:tasks.rados.rados.0.mira097.stdout:748: delete oid 372 current snap is 27 2014-08-22 19:25:56,234.234 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:25:56,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.146118 7f6224d87700 20 client.4189.objecter shutdown clearing up homeless session... 2014-08-22 19:25:56,240.240 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.148497 7f6224d87700 1 librados: shutdown 2014-08-22 19:25:56,246.246 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:25:56,247.247 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:25:56,293.293 INFO:tasks.rados.rados.0.mira097.stdout:748: done (1 left) 2014-08-22 19:25:56,294.294 INFO:tasks.rados.rados.0.mira097.stdout:749: read oid 109 snap -1 2014-08-22 19:25:56,294.294 INFO:tasks.rados.rados.0.mira097.stdout:750: delete oid 137 current snap is 27 2014-08-22 19:25:56,325.325 INFO:tasks.rados.rados.0.mira097.stdout:747: finishing copy_from to mira09725123-246 2014-08-22 19:25:56,325.325 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 246 v 835 (ObjNum 566 snap 20 seq_num 566) dirty exists 2014-08-22 19:25:56,329.329 INFO:tasks.rados.rados.0.mira097.stdout:747: finishing copy_from racing read to mira09725123-246 2014-08-22 19:25:56,341.341 INFO:tasks.rados.rados.0.mira097.stdout:749: expect (ObjNum 108 snap 0 seq_num 108) 2014-08-22 19:25:56,397.397 INFO:tasks.rados.rados.0.mira097.stdout:747: done (2 left) 2014-08-22 19:25:56,398.398 INFO:tasks.rados.rados.0.mira097.stdout:749: done (1 left) 2014-08-22 19:25:56,398.398 INFO:tasks.rados.rados.0.mira097.stdout:750: done (0 left) 2014-08-22 19:25:56,398.398 INFO:tasks.rados.rados.0.mira097.stdout:751: delete oid 82 current snap is 27 2014-08-22 19:25:56,436.436 INFO:tasks.rados.rados.0.mira097.stdout:751: done (0 left) 2014-08-22 19:25:56,437.437 INFO:tasks.rados.rados.0.mira097.stdout:752: rollback oid 477 current snap is 27 2014-08-22 19:25:56,437.437 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 477 to 14 2014-08-22 19:25:56,437.437 INFO:tasks.rados.rados.0.mira097.stdout:753: read oid 325 snap 25 2014-08-22 19:25:56,437.437 INFO:tasks.rados.rados.0.mira097.stdout:754: delete oid 344 current snap is 27 2014-08-22 19:25:56,439.439 INFO:tasks.rados.rados.0.mira097.stdout:754: done (2 left) 2014-08-22 19:25:56,439.439 INFO:tasks.rados.rados.0.mira097.stdout:755: snap_create 2014-08-22 19:25:56,439.439 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:25:56,513.513 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 477 v 977 (ObjNum 476 snap 0 seq_num 476) dirty exists 2014-08-22 19:25:56,513.513 INFO:tasks.rados.rados.0.mira097.stdout:752: done (1 left) 2014-08-22 19:25:56,513.513 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:25:56,545.545 INFO:tasks.rados.rados.0.mira097.stdout:753: expect (ObjNum 324 snap 0 seq_num 324) 2014-08-22 19:25:56,713.713 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.620877 7f3e28e8d700 1 librados: starting msgr at :/0 2014-08-22 19:25:56,714.714 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.620890 7f3e28e8d700 1 librados: starting objecter 2014-08-22 19:25:56,714.714 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.621065 7f3e28e8d700 1 librados: setting wanted keys 2014-08-22 19:25:56,714.714 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.621070 7f3e28e8d700 1 librados: calling monclient init 2014-08-22 19:25:56,714.714 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.623039 7f3e21d8a700 10 client.?.objecter ms_handle_connect 0x7f3e240355e0 2014-08-22 19:25:56,715.715 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.623081 7f3e21d8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:56,718.718 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.626616 7f3e28e8d700 10 client.4192.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:56,719.719 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.626699 7f3e28e8d700 1 librados: init done 2014-08-22 19:25:56,719.719 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627422 7f3e21d8a700 10 client.4192.objecter ms_dispatch 0x7f3e24000f80 osd_map(93..93 src has 1..93) v3 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627429 7f3e21d8a700 3 client.4192.objecter handle_osd_map got epochs [93,93] > 0 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627432 7f3e21d8a700 3 client.4192.objecter handle_osd_map decoding full epoch 93 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627526 7f3e21d8a700 20 client.4192.objecter dump_active .. 0 homeless 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627565 7f3e21d8a700 10 client.4192.objecter ms_dispatch 0x7f3e24000f80 osd_map(93..93 src has 1..93) v3 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627568 7f3e21d8a700 3 client.4192.objecter handle_osd_map ignoring epochs [93,93] <= 93 2014-08-22 19:25:56,720.720 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.627569 7f3e21d8a700 20 client.4192.objecter dump_active .. 0 homeless 2014-08-22 19:25:56,893.893 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:25:57,076.076 INFO:tasks.rados.rados.0.mira097.stdout:753: done (0 left) 2014-08-22 19:25:57,076.076 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.723766 7f3e28e8d700 20 client.4192.objecter shutdown clearing up homeless session... 2014-08-22 19:25:57,076.076 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:49.724285 7f3e28e8d700 1 librados: shutdown 2014-08-22 19:25:57,234.234 INFO:tasks.rados.rados.0.mira097.stdout:755: done (0 left) 2014-08-22 19:25:57,235.235 INFO:tasks.rados.rados.0.mira097.stdout:756: write oid 384 current snap is 28 2014-08-22 19:25:57,235.235 INFO:tasks.rados.rados.0.mira097.stdout:756: seq_num 594 ranges {566084=512483,1835704=483314,2566083=1} 2014-08-22 19:25:57,240.240 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:25:57,257.257 INFO:tasks.rados.rados.0.mira097.stdout:756: writing mira09725123-384 from 566084 to 1078567 tid 1 2014-08-22 19:25:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:756: writing mira09725123-384 from 1835704 to 2319018 tid 2 2014-08-22 19:25:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:756: writing mira09725123-384 from 2566083 to 2566084 tid 3 2014-08-22 19:25:57,279.279 INFO:tasks.rados.rados.0.mira097.stdout:757: read oid 488 snap -1 2014-08-22 19:25:57,280.280 INFO:tasks.rados.rados.0.mira097.stdout:758: read oid 197 snap -1 2014-08-22 19:25:57,280.280 INFO:tasks.rados.rados.0.mira097.stdout:759: rollback oid 59 current snap is 28 2014-08-22 19:25:57,280.280 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 59 to 14 2014-08-22 19:25:57,280.280 INFO:tasks.rados.rados.0.mira097.stdout:760: write oid 102 current snap is 28 2014-08-22 19:25:57,280.280 INFO:tasks.rados.rados.0.mira097.stdout:760: seq_num 595 ranges {509889=767935,1681035=685733,3016151=293738} 2014-08-22 19:25:57,308.308 INFO:tasks.rados.rados.0.mira097.stdout:760: writing mira09725123-102 from 509889 to 1277824 tid 1 2014-08-22 19:25:57,335.335 INFO:tasks.rados.rados.0.mira097.stdout:760: writing mira09725123-102 from 1681035 to 2366768 tid 2 2014-08-22 19:25:57,351.351 INFO:tasks.rados.rados.0.mira097.stdout:760: writing mira09725123-102 from 3016151 to 3309889 tid 3 2014-08-22 19:25:57,356.356 INFO:tasks.rados.rados.0.mira097.stdout:761: snap_remove snap 27 2014-08-22 19:25:57,379.379 INFO:tasks.rados.rados.0.mira097.stdout:760: finishing write tid 1 to mira09725123-102 2014-08-22 19:25:57,396.396 INFO:tasks.rados.rados.0.mira097.stdout:760: finishing write tid 2 to mira09725123-102 2014-08-22 19:25:57,471.471 INFO:tasks.rados.rados.0.mira097.stdout:760: finishing write tid 3 to mira09725123-102 2014-08-22 19:25:57,487.487 INFO:tasks.rados.rados.0.mira097.stdout:756: finishing write tid 1 to mira09725123-384 2014-08-22 19:25:57,488.488 INFO:tasks.rados.rados.0.mira097.stdout:756: finishing write tid 2 to mira09725123-384 2014-08-22 19:25:57,488.488 INFO:tasks.rados.rados.0.mira097.stdout:756: finishing write tid 3 to mira09725123-384 2014-08-22 19:25:57,488.488 INFO:tasks.rados.rados.0.mira097.stdout:760: finishing write tid 5 to mira09725123-102 2014-08-22 19:25:57,489.489 INFO:tasks.rados.rados.0.mira097.stdout:756: finishing write tid 5 to mira09725123-384 2014-08-22 19:25:57,489.489 INFO:tasks.rados.rados.0.mira097.stdout:756: finishing write tid 6 to mira09725123-384 2014-08-22 19:25:57,489.489 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 384 v 987 (ObjNum 594 snap 28 seq_num 594) dirty exists 2014-08-22 19:25:57,496.496 INFO:tasks.rados.rados.0.mira097.stdout:760: finishing write tid 6 to mira09725123-102 2014-08-22 19:25:57,496.496 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 102 v 1145 (ObjNum 595 snap 28 seq_num 595) dirty exists 2014-08-22 19:25:57,529.529 INFO:tasks.rados.rados.0.mira097.stdout:757: expect (ObjNum 487 snap 0 seq_num 487) 2014-08-22 19:25:57,689.689 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.596353 7f61a1700700 1 librados: starting msgr at :/0 2014-08-22 19:25:57,689.689 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.596360 7f61a1700700 1 librados: starting objecter 2014-08-22 19:25:57,689.689 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.596514 7f61a1700700 1 librados: setting wanted keys 2014-08-22 19:25:57,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.596520 7f61a1700700 1 librados: calling monclient init 2014-08-22 19:25:57,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.597382 7f619a58b700 10 client.?.objecter ms_handle_connect 0x7f619c0355e0 2014-08-22 19:25:57,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.597419 7f619a58b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:57,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599329 7f61a1700700 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:57,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599407 7f61a1700700 1 librados: init done 2014-08-22 19:25:57,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599712 7f619a58b700 10 client.4176.objecter ms_dispatch 0x7f619c000f80 osd_map(94..94 src has 1..94) v3 2014-08-22 19:25:57,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599718 7f619a58b700 3 client.4176.objecter handle_osd_map got epochs [94,94] > 0 2014-08-22 19:25:57,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599720 7f619a58b700 3 client.4176.objecter handle_osd_map decoding full epoch 94 2014-08-22 19:25:57,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599804 7f619a58b700 20 client.4176.objecter dump_active .. 0 homeless 2014-08-22 19:25:57,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599834 7f619a58b700 10 client.4176.objecter ms_dispatch 0x7f619c000f80 osd_map(94..94 src has 1..94) v3 2014-08-22 19:25:57,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599838 7f619a58b700 3 client.4176.objecter handle_osd_map ignoring epochs [94,94] <= 94 2014-08-22 19:25:57,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.599839 7f619a58b700 20 client.4176.objecter dump_active .. 0 homeless 2014-08-22 19:25:57,788.788 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:25:58,004.004 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.687103 7f61a1700700 20 client.4176.objecter shutdown clearing up homeless session... 2014-08-22 19:25:58,004.004 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:50.687729 7f61a1700700 1 librados: shutdown 2014-08-22 19:25:58,027.027 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:25:58,521.521 INFO:tasks.rados.rados.0.mira097.stdout:758: expect (ObjNum 196 snap 0 seq_num 196) 2014-08-22 19:25:58,644.644 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.551190 7fd04f30d700 1 librados: starting msgr at :/0 2014-08-22 19:25:58,644.644 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.551200 7fd04f30d700 1 librados: starting objecter 2014-08-22 19:25:58,645.645 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.551370 7fd04f30d700 1 librados: setting wanted keys 2014-08-22 19:25:58,645.645 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.551375 7fd04f30d700 1 librados: calling monclient init 2014-08-22 19:25:58,645.645 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.552048 7fd03ffff700 10 client.?.objecter ms_handle_connect 0x7fd0480355e0 2014-08-22 19:25:58,645.645 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.552069 7fd03ffff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:58,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553323 7fd04f30d700 10 client.4179.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:58,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553432 7fd04f30d700 1 librados: init done 2014-08-22 19:25:58,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553689 7fd03ffff700 10 client.4179.objecter ms_dispatch 0x7fd048000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553694 7fd03ffff700 3 client.4179.objecter handle_osd_map got epochs [95,95] > 0 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553696 7fd03ffff700 3 client.4179.objecter handle_osd_map decoding full epoch 95 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553764 7fd03ffff700 20 client.4179.objecter dump_active .. 0 homeless 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553799 7fd03ffff700 10 client.4179.objecter ms_dispatch 0x7fd048000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553803 7fd03ffff700 3 client.4179.objecter handle_osd_map ignoring epochs [95,95] <= 95 2014-08-22 19:25:58,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.553806 7fd03ffff700 20 client.4179.objecter dump_active .. 0 homeless 2014-08-22 19:25:58,726.726 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.633366 7fd04f30d700 20 client.4179.objecter shutdown clearing up homeless session... 2014-08-22 19:25:58,727.727 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:51.633784 7fd04f30d700 1 librados: shutdown 2014-08-22 19:25:58,729.729 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003786', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001834', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:25:35.667509', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873981888, u'health': u'HEALTH_OK', u'kb_used': 22799164, u'store_stats': {u'bytes_total': 5250095, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.468804', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 877079888, u'health': u'HEALTH_OK', u'kb_used': 19701164, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.072747', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873980544, u'health': u'HEALTH_OK', u'kb_used': 22800508, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 54 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.a has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 62, u'data_bytes': 1263528598, u'bytes_used': 3413446656, u'op_per_sec': 74, u'version': 158, u'pgs_by_state': [{u'count': 62, u'state_name': u'active+clean'}], u'read_bytes_sec': 9309330, u'write_bytes_sec': 20211191, u'bytes_avail': 5994885599232}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:25:58,730.730 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:25:58,731.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:25:58,731.731 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:25:58,731.731 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:25:59,211.211 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.118822 7f58f7986700 1 librados: starting msgr at :/0 2014-08-22 19:25:59,212.212 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.118831 7f58f7986700 1 librados: starting objecter 2014-08-22 19:25:59,212.212 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.118985 7f58f7986700 1 librados: setting wanted keys 2014-08-22 19:25:59,212.212 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.118999 7f58f7986700 1 librados: calling monclient init 2014-08-22 19:25:59,212.212 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.119791 7f58f4a33700 10 client.?.objecter ms_handle_connect 0x7f58f00355e0 2014-08-22 19:25:59,212.212 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.119820 7f58f4a33700 10 client.?.objecter resend_mon_ops 2014-08-22 19:25:59,213.213 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121095 7f58f7986700 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121195 7f58f7986700 1 librados: init done 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121458 7f58f4a33700 10 client.4175.objecter ms_dispatch 0x7f58f0000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121464 7f58f4a33700 3 client.4175.objecter handle_osd_map got epochs [95,95] > 0 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121466 7f58f4a33700 3 client.4175.objecter handle_osd_map decoding full epoch 95 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121565 7f58f4a33700 20 client.4175.objecter dump_active .. 0 homeless 2014-08-22 19:25:59,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121646 7f58f4a33700 10 client.4175.objecter ms_dispatch 0x7f58f0000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:25:59,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121651 7f58f4a33700 3 client.4175.objecter handle_osd_map ignoring epochs [95,95] <= 95 2014-08-22 19:25:59,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.121653 7f58f4a33700 20 client.4175.objecter dump_active .. 0 homeless 2014-08-22 19:25:59,314.314 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:25:59,498.498 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.212992 7f58f7986700 20 client.4175.objecter shutdown clearing up homeless session... 2014-08-22 19:25:59,499.499 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:52.213377 7f58f7986700 1 librados: shutdown 2014-08-22 19:25:59,522.522 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:25:59,522.522 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 64' 2014-08-22 19:25:59,839.839 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 59 v 1071 (ObjNum 58 snap 0 seq_num 58) dirty exists 2014-08-22 19:25:59,839.839 INFO:tasks.rados.rados.0.mira097.stdout:756: done (5 left) 2014-08-22 19:25:59,839.839 INFO:tasks.rados.rados.0.mira097.stdout:757: done (4 left) 2014-08-22 19:25:59,839.839 INFO:tasks.rados.rados.0.mira097.stdout:758: done (3 left) 2014-08-22 19:25:59,839.839 INFO:tasks.rados.rados.0.mira097.stdout:759: done (2 left) 2014-08-22 19:25:59,840.840 INFO:tasks.rados.rados.0.mira097.stdout:760: done (1 left) 2014-08-22 19:25:59,840.840 INFO:tasks.rados.rados.0.mira097.stdout:761: done (0 left) 2014-08-22 19:25:59,840.840 INFO:tasks.rados.rados.0.mira097.stdout:762: copy_from oid 186 from oid 404 current snap is 28 2014-08-22 19:25:59,841.841 INFO:tasks.rados.rados.0.mira097.stdout:763: copy_from oid 100 from oid 30 current snap is 28 2014-08-22 19:25:59,841.841 INFO:tasks.rados.rados.0.mira097.stdout:764: read oid 107 snap -1 2014-08-22 19:25:59,841.841 INFO:tasks.rados.rados.0.mira097.stdout:765: write oid 129 current snap is 28 2014-08-22 19:25:59,841.841 INFO:tasks.rados.rados.0.mira097.stdout:765: seq_num 596 ranges {722495=491163,1662943=507352,2685152=549802,3522494=1} 2014-08-22 19:25:59,869.869 INFO:tasks.rados.rados.0.mira097.stdout:765: writing mira09725123-129 from 722495 to 1213658 tid 1 2014-08-22 19:25:59,884.884 INFO:tasks.rados.rados.0.mira097.stdout:765: writing mira09725123-129 from 1662943 to 2170295 tid 2 2014-08-22 19:25:59,903.903 INFO:tasks.rados.rados.0.mira097.stdout:765: writing mira09725123-129 from 2685152 to 3234954 tid 3 2014-08-22 19:25:59,906.906 INFO:tasks.rados.rados.0.mira097.stdout:765: writing mira09725123-129 from 3522494 to 3522495 tid 4 2014-08-22 19:25:59,907.907 INFO:tasks.rados.rados.0.mira097.stdout:766: write oid 300 current snap is 28 2014-08-22 19:25:59,907.907 INFO:tasks.rados.rados.0.mira097.stdout:766: seq_num 597 ranges {266300=1} 2014-08-22 19:25:59,908.908 INFO:tasks.rados.rados.0.mira097.stdout:766: writing mira09725123-300 from 266300 to 266301 tid 1 2014-08-22 19:25:59,908.908 INFO:tasks.rados.rados.0.mira097.stdout:767: read oid 279 snap 20 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:768: copy_from oid 167 from oid 346 current snap is 28 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:769: read oid 68 snap -1 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:770: copy_from oid 400 from oid 412 current snap is 28 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:771: read oid 149 snap -1 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:772: write oid 433 current snap is 28 2014-08-22 19:25:59,910.910 INFO:tasks.rados.rados.0.mira097.stdout:772: seq_num 598 ranges {610106=400000} 2014-08-22 19:25:59,928.928 INFO:tasks.rados.rados.0.mira097.stdout:772: writing mira09725123-433 from 610106 to 1010106 tid 1 2014-08-22 19:25:59,929.929 INFO:tasks.rados.rados.0.mira097.stdout:773: write oid 109 current snap is 28 2014-08-22 19:25:59,929.929 INFO:tasks.rados.rados.0.mira097.stdout:773: seq_num 599 ranges {706359=1} 2014-08-22 19:25:59,936.936 INFO:tasks.rados.rados.0.mira097.stdout:773: writing mira09725123-109 from 706359 to 706360 tid 1 2014-08-22 19:25:59,937.937 INFO:tasks.rados.rados.0.mira097.stdout:774: write oid 133 current snap is 28 2014-08-22 19:25:59,937.937 INFO:tasks.rados.rados.0.mira097.stdout:774: seq_num 600 ranges {517942=400000} 2014-08-22 19:25:59,955.955 INFO:tasks.rados.rados.0.mira097.stdout:774: writing mira09725123-133 from 517942 to 917942 tid 1 2014-08-22 19:25:59,957.957 INFO:tasks.rados.rados.0.mira097.stdout:775: delete oid 213 current snap is 28 2014-08-22 19:25:59,957.957 INFO:tasks.rados.rados.0.mira097.stdout:771: expect (ObjNum 546 snap 12 seq_num 546) 2014-08-22 19:26:00,154.154 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.061351 7f9354230700 1 librados: starting msgr at :/0 2014-08-22 19:26:00,155.155 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.061362 7f9354230700 1 librados: starting objecter 2014-08-22 19:26:00,155.155 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.061504 7f9354230700 1 librados: setting wanted keys 2014-08-22 19:26:00,155.155 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.061521 7f9354230700 1 librados: calling monclient init 2014-08-22 19:26:00,155.155 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.062261 7f93512dd700 10 client.?.objecter ms_handle_connect 0x7f934c0355e0 2014-08-22 19:26:00,155.155 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.062294 7f93512dd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:00,157.157 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063487 7f9354230700 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:00,157.157 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063598 7f9354230700 1 librados: init done 2014-08-22 19:26:00,157.157 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063884 7f93512dd700 10 client.4178.objecter ms_dispatch 0x7f934c000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:26:00,157.157 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063893 7f93512dd700 3 client.4178.objecter handle_osd_map got epochs [95,95] > 0 2014-08-22 19:26:00,157.157 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063896 7f93512dd700 3 client.4178.objecter handle_osd_map decoding full epoch 95 2014-08-22 19:26:00,158.158 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.063982 7f93512dd700 20 client.4178.objecter dump_active .. 0 homeless 2014-08-22 19:26:00,158.158 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.064023 7f93512dd700 10 client.4178.objecter ms_dispatch 0x7f934c000f80 osd_map(95..95 src has 1..95) v3 2014-08-22 19:26:00,158.158 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.064028 7f93512dd700 3 client.4178.objecter handle_osd_map ignoring epochs [95,95] <= 95 2014-08-22 19:26:00,158.158 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.064031 7f93512dd700 20 client.4178.objecter dump_active .. 0 homeless 2014-08-22 19:26:00,268.268 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 64 2014-08-22 19:26:00,275.275 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.180894 7f9354230700 20 client.4178.objecter shutdown clearing up homeless session... 2014-08-22 19:26:00,275.275 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:53.181338 7f9354230700 1 librados: shutdown 2014-08-22 19:26:01,376.376 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 1 to mira09725123-129 2014-08-22 19:26:01,377.377 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 2 to mira09725123-129 2014-08-22 19:26:01,377.377 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 3 to mira09725123-129 2014-08-22 19:26:01,377.377 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 4 to mira09725123-129 2014-08-22 19:26:01,377.377 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 6 to mira09725123-129 2014-08-22 19:26:01,377.377 INFO:tasks.rados.rados.0.mira097.stdout:766: finishing write tid 1 to mira09725123-300 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:766: finishing write tid 3 to mira09725123-300 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:771: done (13 left) 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:775: done (12 left) 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:776: delete oid 269 current snap is 28 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:762: finishing copy_from to mira09725123-186 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 186 v 540 (ObjNum 403 snap 0 seq_num 403) dirty exists 2014-08-22 19:26:01,378.378 INFO:tasks.rados.rados.0.mira097.stdout:764: expect (ObjNum 106 snap 0 seq_num 106) 2014-08-22 19:26:03,229.229 INFO:tasks.rados.rados.0.mira097.stdout:773: finishing write tid 1 to mira09725123-109 2014-08-22 19:26:03,229.229 INFO:tasks.rados.rados.0.mira097.stdout:773: finishing write tid 3 to mira09725123-109 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:773: finishing write tid 4 to mira09725123-109 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 109 v 1081 (ObjNum 599 snap 28 seq_num 599) dirty exists 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:770: finishing copy_from to mira09725123-400 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 400 v 820 (ObjNum 523 snap 8 seq_num 523) dirty exists 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:770: finishing copy_from racing read to mira09725123-400 2014-08-22 19:26:03,230.230 INFO:tasks.rados.rados.0.mira097.stdout:765: finishing write tid 7 to mira09725123-129 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 129 v 1159 (ObjNum 596 snap 28 seq_num 596) dirty exists 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:772: finishing write tid 1 to mira09725123-433 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:774: finishing write tid 1 to mira09725123-133 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:772: finishing write tid 3 to mira09725123-433 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:774: finishing write tid 3 to mira09725123-133 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:762: finishing copy_from racing read to mira09725123-186 2014-08-22 19:26:03,231.231 INFO:tasks.rados.rados.0.mira097.stdout:774: finishing write tid 4 to mira09725123-133 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 133 v 1003 (ObjNum 600 snap 28 seq_num 600) dirty exists 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:766: finishing write tid 4 to mira09725123-300 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 300 v 1163 (ObjNum 597 snap 28 seq_num 597) dirty exists 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:772: finishing write tid 4 to mira09725123-433 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 433 v 1166 (ObjNum 598 snap 28 seq_num 598) dirty exists 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:768: finishing copy_from to mira09725123-167 2014-08-22 19:26:03,232.232 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 167 v 453 (ObjNum 345 snap 0 seq_num 345) dirty exists 2014-08-22 19:26:03,233.233 INFO:tasks.rados.rados.0.mira097.stdout:767: expect (ObjNum 278 snap 0 seq_num 278) 2014-08-22 19:26:04,899.899 INFO:tasks.rados.rados.0.mira097.stdout:763: finishing copy_from to mira09725123-100 2014-08-22 19:26:04,899.899 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 100 v 29 (ObjNum 29 snap 0 seq_num 29) dirty exists 2014-08-22 19:26:04,899.899 INFO:tasks.rados.rados.0.mira097.stdout:768: finishing copy_from racing read to mira09725123-167 2014-08-22 19:26:04,899.899 INFO:tasks.rados.rados.0.mira097.stdout:763: finishing copy_from racing read to mira09725123-100 2014-08-22 19:26:04,899.899 INFO:tasks.rados.rados.0.mira097.stdout:769: expect (ObjNum 67 snap 0 seq_num 67) 2014-08-22 19:26:05,280.280 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:05,280.280 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:05,281.281 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:26:05,281.281 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:05,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.634973 7f9c1d11a700 1 librados: starting msgr at :/0 2014-08-22 19:26:05,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.634978 7f9c1d11a700 1 librados: starting objecter 2014-08-22 19:26:05,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.635184 7f9c1d11a700 1 librados: setting wanted keys 2014-08-22 19:26:05,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.635189 7f9c1d11a700 1 librados: calling monclient init 2014-08-22 19:26:05,729.729 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.636910 7f9c160ae700 10 client.?.objecter ms_handle_connect 0x7f9c180355e0 2014-08-22 19:26:05,729.729 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.636931 7f9c160ae700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:05,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.640025 7f9c1d11a700 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:05,732.732 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.640127 7f9c1d11a700 1 librados: init done 2014-08-22 19:26:05,733.733 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.640935 7f9c160ae700 10 client.4195.objecter ms_dispatch 0x7f9c18000f80 osd_map(97..97 src has 1..97) v3 2014-08-22 19:26:05,733.733 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.640941 7f9c160ae700 3 client.4195.objecter handle_osd_map got epochs [97,97] > 0 2014-08-22 19:26:05,733.733 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.640943 7f9c160ae700 3 client.4195.objecter handle_osd_map decoding full epoch 97 2014-08-22 19:26:05,733.733 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.641011 7f9c160ae700 20 client.4195.objecter dump_active .. 0 homeless 2014-08-22 19:26:05,830.830 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.641165 7f9c160ae700 10 client.4195.objecter ms_dispatch 0x7f9c18000f80 osd_map(97..97 src has 1..97) v3 2014-08-22 19:26:05,831.831 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.641167 7f9c160ae700 3 client.4195.objecter handle_osd_map ignoring epochs [97,97] <= 97 2014-08-22 19:26:05,831.831 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.641169 7f9c160ae700 20 client.4195.objecter dump_active .. 0 homeless 2014-08-22 19:26:05,912.912 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:06,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.741334 7f9c1d11a700 20 client.4195.objecter shutdown clearing up homeless session... 2014-08-22 19:26:06,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:25:58.741981 7f9c1d11a700 1 librados: shutdown 2014-08-22 19:26:06,540.540 INFO:tasks.rados.rados.0.mira097.stdout:762: done (12 left) 2014-08-22 19:26:06,540.540 INFO:tasks.rados.rados.0.mira097.stdout:763: done (11 left) 2014-08-22 19:26:06,540.540 INFO:tasks.rados.rados.0.mira097.stdout:764: done (10 left) 2014-08-22 19:26:06,541.541 INFO:tasks.rados.rados.0.mira097.stdout:765: done (9 left) 2014-08-22 19:26:06,541.541 INFO:tasks.rados.rados.0.mira097.stdout:766: done (8 left) 2014-08-22 19:26:06,541.541 INFO:tasks.rados.rados.0.mira097.stdout:767: done (7 left) 2014-08-22 19:26:06,541.541 INFO:tasks.rados.rados.0.mira097.stdout:768: done (6 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:769: done (5 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:770: done (4 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:772: done (3 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:773: done (2 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:774: done (1 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:776: done (0 left) 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:777: rollback oid 296 current snap is 28 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 296 to 25 2014-08-22 19:26:06,542.542 INFO:tasks.rados.rados.0.mira097.stdout:778: delete oid 98 current snap is 28 2014-08-22 19:26:06,561.561 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 296 v 1099 (ObjNum 295 snap 0 seq_num 295) dirty exists 2014-08-22 19:26:06,597.597 INFO:tasks.rados.rados.0.mira097.stdout:777: done (1 left) 2014-08-22 19:26:06,598.598 INFO:tasks.rados.rados.0.mira097.stdout:778: done (0 left) 2014-08-22 19:26:06,598.598 INFO:tasks.rados.rados.0.mira097.stdout:779: delete oid 177 current snap is 28 2014-08-22 19:26:06,676.676 INFO:tasks.rados.rados.0.mira097.stdout:779: done (0 left) 2014-08-22 19:26:06,676.676 INFO:tasks.rados.rados.0.mira097.stdout:780: write oid 18 current snap is 28 2014-08-22 19:26:06,677.677 INFO:tasks.rados.rados.0.mira097.stdout:780: seq_num 605 ranges {107818=1} 2014-08-22 19:26:06,677.677 INFO:tasks.rados.rados.0.mira097.stdout:780: writing mira09725123-18 from 107818 to 107819 tid 1 2014-08-22 19:26:06,678.678 INFO:tasks.rados.rados.0.mira097.stdout:781: read oid 169 snap -1 2014-08-22 19:26:06,678.678 INFO:tasks.rados.rados.0.mira097.stdout:782: snap_remove snap 14 2014-08-22 19:26:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:780: finishing write tid 1 to mira09725123-18 2014-08-22 19:26:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:780: finishing write tid 3 to mira09725123-18 2014-08-22 19:26:06,698.698 INFO:tasks.rados.rados.0.mira097.stdout:780: finishing write tid 4 to mira09725123-18 2014-08-22 19:26:06,698.698 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 18 v 1016 (ObjNum 605 snap 28 seq_num 605) dirty exists 2014-08-22 19:26:06,980.980 INFO:tasks.rados.rados.0.mira097.stdout:781: expect (ObjNum 168 snap 0 seq_num 168) 2014-08-22 19:26:07,627.627 INFO:tasks.rados.rados.0.mira097.stdout:780: done (2 left) 2014-08-22 19:26:07,627.627 INFO:tasks.rados.rados.0.mira097.stdout:781: done (1 left) 2014-08-22 19:26:07,628.628 INFO:tasks.rados.rados.0.mira097.stdout:782: done (0 left) 2014-08-22 19:26:07,628.628 INFO:tasks.rados.rados.0.mira097.stdout:783: read oid 211 snap 20 2014-08-22 19:26:07,628.628 INFO:tasks.rados.rados.0.mira097.stdout:784: copy_from oid 44 from oid 198 current snap is 28 2014-08-22 19:26:07,628.628 INFO:tasks.rados.rados.0.mira097.stdout:785: snap_remove snap 25 2014-08-22 19:26:07,693.693 INFO:tasks.rados.rados.0.mira097.stdout:783: expect (ObjNum 503 snap 3 seq_num 503) 2014-08-22 19:26:08,615.615 INFO:tasks.rados.rados.0.mira097.stdout:784: finishing copy_from to mira09725123-44 2014-08-22 19:26:08,615.615 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 44 v 241 (ObjNum 207 snap 0 seq_num 207) dirty exists 2014-08-22 19:26:08,616.616 INFO:tasks.rados.rados.0.mira097.stdout:784: finishing copy_from racing read to mira09725123-44 2014-08-22 19:26:08,633.633 INFO:tasks.rados.rados.0.mira097.stdout:783: done (2 left) 2014-08-22 19:26:08,634.634 INFO:tasks.rados.rados.0.mira097.stdout:784: done (1 left) 2014-08-22 19:26:08,634.634 INFO:tasks.rados.rados.0.mira097.stdout:785: done (0 left) 2014-08-22 19:26:08,634.634 INFO:tasks.rados.rados.0.mira097.stdout:786: read oid 329 snap 23 2014-08-22 19:26:08,634.634 INFO:tasks.rados.rados.0.mira097.stdout:787: write oid 274 current snap is 28 2014-08-22 19:26:08,634.634 INFO:tasks.rados.rados.0.mira097.stdout:787: seq_num 606 ranges {603048=585271,1707433=747224,2889662=783595,3803047=1} 2014-08-22 19:26:08,666.666 INFO:tasks.rados.rados.0.mira097.stdout:787: writing mira09725123-274 from 603048 to 1188319 tid 1 2014-08-22 19:26:08,695.695 INFO:tasks.rados.rados.0.mira097.stdout:787: writing mira09725123-274 from 1707433 to 2454657 tid 2 2014-08-22 19:26:08,718.718 INFO:tasks.rados.rados.0.mira097.stdout:787: writing mira09725123-274 from 2889662 to 3673257 tid 3 2014-08-22 19:26:08,721.721 INFO:tasks.rados.rados.0.mira097.stdout:787: writing mira09725123-274 from 3803047 to 3803048 tid 4 2014-08-22 19:26:08,722.722 INFO:tasks.rados.rados.0.mira097.stdout:788: read oid 40 snap -1 2014-08-22 19:26:08,722.722 INFO:tasks.rados.rados.0.mira097.stdout:789: delete oid 219 current snap is 28 2014-08-22 19:26:08,744.744 INFO:tasks.rados.rados.0.mira097.stdout:786: expect (ObjNum 328 snap 0 seq_num 328) 2014-08-22 19:26:10,446.446 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 1 to mira09725123-274 2014-08-22 19:26:10,446.446 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 2 to mira09725123-274 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 3 to mira09725123-274 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 4 to mira09725123-274 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 6 to mira09725123-274 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:787: finishing write tid 7 to mira09725123-274 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 274 v 1140 (ObjNum 606 snap 28 seq_num 606) dirty exists 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:786: done (3 left) 2014-08-22 19:26:10,447.447 INFO:tasks.rados.rados.0.mira097.stdout:787: done (2 left) 2014-08-22 19:26:10,448.448 INFO:tasks.rados.rados.0.mira097.stdout:789: done (1 left) 2014-08-22 19:26:10,448.448 INFO:tasks.rados.rados.0.mira097.stdout:790: copy_from oid 229 from oid 414 current snap is 28 2014-08-22 19:26:10,448.448 INFO:tasks.rados.rados.0.mira097.stdout:788: expect (ObjNum 39 snap 0 seq_num 39) 2014-08-22 19:26:11,278.278 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:11,278.278 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:26:11,699.699 INFO:tasks.rados.rados.0.mira097.stdout:790: finishing copy_from to mira09725123-229 2014-08-22 19:26:11,700.700 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 229 v 584 (ObjNum 413 snap 0 seq_num 413) dirty exists 2014-08-22 19:26:11,700.700 INFO:tasks.rados.rados.0.mira097.stdout:790: finishing copy_from racing read to mira09725123-229 2014-08-22 19:26:11,700.700 INFO:tasks.rados.rados.0.mira097.stdout:788: done (1 left) 2014-08-22 19:26:11,700.700 INFO:tasks.rados.rados.0.mira097.stdout:790: done (0 left) 2014-08-22 19:26:11,700.700 INFO:tasks.rados.rados.0.mira097.stdout:791: snap_create 2014-08-22 19:26:11,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.650599 7f8fb3950700 1 librados: starting msgr at :/0 2014-08-22 19:26:11,744.744 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.650607 7f8fb3950700 1 librados: starting objecter 2014-08-22 19:26:11,744.744 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.650808 7f8fb3950700 1 librados: setting wanted keys 2014-08-22 19:26:11,744.744 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.650818 7f8fb3950700 1 librados: calling monclient init 2014-08-22 19:26:11,744.744 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.652412 7f8fb09fd700 10 client.?.objecter ms_handle_connect 0x7f8fac0355e0 2014-08-22 19:26:11,744.744 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.652451 7f8fb09fd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:11,747.747 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.655548 7f8fb3950700 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:11,748.748 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.655736 7f8fb3950700 1 librados: init done 2014-08-22 19:26:11,748.748 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656276 7f8fb09fd700 10 client.4182.objecter ms_dispatch 0x7f8fac000f80 osd_map(99..99 src has 1..99) v3 2014-08-22 19:26:11,829.829 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656287 7f8fb09fd700 3 client.4182.objecter handle_osd_map got epochs [99,99] > 0 2014-08-22 19:26:11,829.829 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656289 7f8fb09fd700 3 client.4182.objecter handle_osd_map decoding full epoch 99 2014-08-22 19:26:11,829.829 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656397 7f8fb09fd700 20 client.4182.objecter dump_active .. 0 homeless 2014-08-22 19:26:11,830.830 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656605 7f8fb09fd700 10 client.4182.objecter ms_dispatch 0x7f8fac000f80 osd_map(99..99 src has 1..99) v3 2014-08-22 19:26:11,830.830 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656614 7f8fb09fd700 3 client.4182.objecter handle_osd_map ignoring epochs [99,99] <= 99 2014-08-22 19:26:11,830.830 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:04.656618 7f8fb09fd700 20 client.4182.objecter dump_active .. 0 homeless 2014-08-22 19:26:12,656.656 INFO:tasks.rados.rados.0.mira097.stdout:791: done (0 left) 2014-08-22 19:26:12,656.656 INFO:tasks.rados.rados.0.mira097.stdout:792: copy_from oid 321 from oid 317 current snap is 29 2014-08-22 19:26:12,656.656 INFO:tasks.rados.rados.0.mira097.stdout:793: read oid 222 snap -1 2014-08-22 19:26:12,656.656 INFO:tasks.rados.rados.0.mira097.stdout:794: delete oid 273 current snap is 29 2014-08-22 19:26:12,695.695 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:26:12,701.701 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:05.608019 7f8fb3950700 20 client.4182.objecter shutdown clearing up homeless session... 2014-08-22 19:26:12,702.702 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:05.608665 7f8fb3950700 1 librados: shutdown 2014-08-22 19:26:12,710.710 INFO:tasks.rados.rados.0.mira097.stdout:794: done (2 left) 2014-08-22 19:26:12,710.710 INFO:tasks.rados.rados.0.mira097.stdout:795: delete oid 472 current snap is 29 2014-08-22 19:26:12,710.710 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:26:12,755.755 INFO:tasks.rados.rados.0.mira097.stdout:793: expect (ObjNum 221 snap 0 seq_num 221) 2014-08-22 19:26:13,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.082381 7f51c673f700 1 librados: starting msgr at :/0 2014-08-22 19:26:13,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.082387 7f51c673f700 1 librados: starting objecter 2014-08-22 19:26:13,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.082842 7f51c673f700 1 librados: setting wanted keys 2014-08-22 19:26:13,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.082847 7f51c673f700 1 librados: calling monclient init 2014-08-22 19:26:13,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.084574 7f51beffd700 10 client.?.objecter ms_handle_connect 0x7f51c00355e0 2014-08-22 19:26:13,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:06.084609 7f51beffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:14,314.314 INFO:tasks.rados.rados.0.mira097.stdout:792: finishing copy_from to mira09725123-321 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 321 v 445 (ObjNum 316 snap 0 seq_num 316) dirty exists 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:792: finishing copy_from racing read to mira09725123-321 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:792: done (2 left) 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:793: done (1 left) 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:795: done (0 left) 2014-08-22 19:26:14,315.315 INFO:tasks.rados.rados.0.mira097.stdout:796: rollback oid 255 current snap is 29 2014-08-22 19:26:14,316.316 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 255 to 20 2014-08-22 19:26:14,316.316 INFO:tasks.rados.rados.0.mira097.stdout:797: write oid 347 current snap is 29 2014-08-22 19:26:14,316.316 INFO:tasks.rados.rados.0.mira097.stdout:797: seq_num 610 ranges {497695=508584,1488121=530853,2539492=673413,3297694=1} 2014-08-22 19:26:14,339.339 INFO:tasks.rados.rados.0.mira097.stdout:797: writing mira09725123-347 from 497695 to 1006279 tid 1 2014-08-22 19:26:14,358.358 INFO:tasks.rados.rados.0.mira097.stdout:797: writing mira09725123-347 from 1488121 to 2018974 tid 2 2014-08-22 19:26:14,382.382 INFO:tasks.rados.rados.0.mira097.stdout:797: writing mira09725123-347 from 2539492 to 3212905 tid 3 2014-08-22 19:26:14,385.385 INFO:tasks.rados.rados.0.mira097.stdout:797: writing mira09725123-347 from 3297694 to 3297695 tid 4 2014-08-22 19:26:14,389.389 INFO:tasks.rados.rados.0.mira097.stdout:798: read oid 437 snap -1 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 255 v 1065 (ObjNum 568 snap 20 seq_num 568) dirty exists 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 1 to mira09725123-347 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 2 to mira09725123-347 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:796: done (2 left) 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:799: rollback oid 218 current snap is 29 2014-08-22 19:26:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 218 to 26 2014-08-22 19:26:14,391.391 INFO:tasks.rados.rados.0.mira097.stdout:800: write oid 377 current snap is 29 2014-08-22 19:26:14,391.391 INFO:tasks.rados.rados.0.mira097.stdout:800: seq_num 611 ranges {592925=765061,1885902=601048,2992924=1} 2014-08-22 19:26:14,410.410 INFO:tasks.rados.rados.0.mira097.stdout:800: writing mira09725123-377 from 592925 to 1357986 tid 1 2014-08-22 19:26:14,429.429 INFO:tasks.rados.rados.0.mira097.stdout:800: writing mira09725123-377 from 1885902 to 2486950 tid 2 2014-08-22 19:26:14,488.488 INFO:tasks.rados.rados.0.mira097.stdout:800: writing mira09725123-377 from 2992924 to 2992925 tid 3 2014-08-22 19:26:14,488.488 INFO:tasks.rados.rados.0.mira097.stdout:801: delete oid 346 current snap is 29 2014-08-22 19:26:14,488.488 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 3 to mira09725123-347 2014-08-22 19:26:14,488.488 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 4 to mira09725123-347 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 218 v 1219 (ObjNum 217 snap 0 seq_num 217) dirty exists 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 6 to mira09725123-347 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:797: finishing write tid 7 to mira09725123-347 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 347 v 1156 (ObjNum 610 snap 29 seq_num 610) dirty exists 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:798: expect (ObjNum 436 snap 0 seq_num 436) 2014-08-22 19:26:14,489.489 INFO:tasks.rados.rados.0.mira097.stdout:800: finishing write tid 1 to mira09725123-377 2014-08-22 19:26:14,537.537 INFO:tasks.rados.rados.0.mira097.stdout:800: finishing write tid 2 to mira09725123-377 2014-08-22 19:26:14,537.537 INFO:tasks.rados.rados.0.mira097.stdout:800: finishing write tid 3 to mira09725123-377 2014-08-22 19:26:14,537.537 INFO:tasks.rados.rados.0.mira097.stdout:800: finishing write tid 5 to mira09725123-377 2014-08-22 19:26:14,573.573 INFO:tasks.rados.rados.0.mira097.stdout:797: done (4 left) 2014-08-22 19:26:14,573.573 INFO:tasks.rados.rados.0.mira097.stdout:798: done (3 left) 2014-08-22 19:26:14,573.573 INFO:tasks.rados.rados.0.mira097.stdout:799: done (2 left) 2014-08-22 19:26:14,573.573 INFO:tasks.rados.rados.0.mira097.stdout:801: done (1 left) 2014-08-22 19:26:14,573.573 INFO:tasks.rados.rados.0.mira097.stdout:802: write oid 316 current snap is 29 2014-08-22 19:26:14,574.574 INFO:tasks.rados.rados.0.mira097.stdout:802: seq_num 613 ranges {631961=460093,1677046=775487,2908417=523544} 2014-08-22 19:26:14,599.599 INFO:tasks.rados.rados.0.mira097.stdout:802: writing mira09725123-316 from 631961 to 1092054 tid 1 2014-08-22 19:26:14,627.627 INFO:tasks.rados.rados.0.mira097.stdout:802: writing mira09725123-316 from 1677046 to 2452533 tid 2 2014-08-22 19:26:14,649.649 INFO:tasks.rados.rados.0.mira097.stdout:802: writing mira09725123-316 from 2908417 to 3431961 tid 3 2014-08-22 19:26:14,650.650 INFO:tasks.rados.rados.0.mira097.stdout:803: read oid 375 snap -1 2014-08-22 19:26:14,650.650 INFO:tasks.rados.rados.0.mira097.stdout:800: finishing write tid 6 to mira09725123-377 2014-08-22 19:26:14,650.650 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 377 v 1162 (ObjNum 611 snap 29 seq_num 611) dirty exists 2014-08-22 19:26:14,650.650 INFO:tasks.rados.rados.0.mira097.stdout:800: done (2 left) 2014-08-22 19:26:14,650.650 INFO:tasks.rados.rados.0.mira097.stdout:804: delete oid 141 current snap is 29 2014-08-22 19:26:14,677.677 INFO:tasks.rados.rados.0.mira097.stdout:802: finishing write tid 1 to mira09725123-316 2014-08-22 19:26:14,682.682 INFO:tasks.rados.rados.0.mira097.stdout:802: finishing write tid 2 to mira09725123-316 2014-08-22 19:26:14,688.688 INFO:tasks.rados.rados.0.mira097.stdout:802: finishing write tid 3 to mira09725123-316 2014-08-22 19:26:14,690.690 INFO:tasks.rados.rados.0.mira097.stdout:802: finishing write tid 5 to mira09725123-316 2014-08-22 19:26:14,691.691 INFO:tasks.rados.rados.0.mira097.stdout:802: finishing write tid 6 to mira09725123-316 2014-08-22 19:26:14,691.691 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 316 v 1075 (ObjNum 613 snap 29 seq_num 613) dirty exists 2014-08-22 19:26:14,708.708 INFO:tasks.rados.rados.0.mira097.stdout:803: expect (ObjNum 423 snap 0 seq_num 423) 2014-08-22 19:26:15,579.579 INFO:tasks.rados.rados.0.mira097.stdout:802: done (2 left) 2014-08-22 19:26:15,579.579 INFO:tasks.rados.rados.0.mira097.stdout:803: done (1 left) 2014-08-22 19:26:15,579.579 INFO:tasks.rados.rados.0.mira097.stdout:804: done (0 left) 2014-08-22 19:26:15,580.580 INFO:tasks.rados.rados.0.mira097.stdout:805: read oid 254 snap -1 2014-08-22 19:26:15,580.580 INFO:tasks.rados.rados.0.mira097.stdout:806: rollback oid 289 current snap is 29 2014-08-22 19:26:15,580.580 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 289 to 23 2014-08-22 19:26:15,580.580 INFO:tasks.rados.rados.0.mira097.stdout:807: rollback oid 436 current snap is 29 2014-08-22 19:26:15,580.580 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 436 to 28 2014-08-22 19:26:15,581.581 INFO:tasks.rados.rados.0.mira097.stdout:808: write oid 456 current snap is 29 2014-08-22 19:26:15,581.581 INFO:tasks.rados.rados.0.mira097.stdout:808: seq_num 615 ranges {389395=1} 2014-08-22 19:26:15,586.586 INFO:tasks.rados.rados.0.mira097.stdout:808: writing mira09725123-456 from 389395 to 389396 tid 1 2014-08-22 19:26:15,586.586 INFO:tasks.rados.rados.0.mira097.stdout:809: write oid 313 current snap is 29 2014-08-22 19:26:15,586.586 INFO:tasks.rados.rados.0.mira097.stdout:809: seq_num 616 ranges {733202=400000} 2014-08-22 19:26:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:809: writing mira09725123-313 from 733202 to 1133202 tid 1 2014-08-22 19:26:15,753.753 INFO:tasks.rados.rados.0.mira097.stdout:810: snap_create 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout:808: finishing write tid 1 to mira09725123-456 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout:808: finishing write tid 3 to mira09725123-456 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout:808: finishing write tid 4 to mira09725123-456 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 456 v 1078 (ObjNum 615 snap 29 seq_num 615) dirty exists 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout:808: done (4 left) 2014-08-22 19:26:15,754.754 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 436 v 1150 (ObjNum 435 snap 0 seq_num 435) dirty exists 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout:807: done (3 left) 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 289 v 0 (ObjNum 2 snap 0 seq_num 0) dirty dne 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout:806: done (2 left) 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:15,755.755 INFO:tasks.rados.rados.0.mira097.stdout:809: finishing write tid 1 to mira09725123-313 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout:809: finishing write tid 3 to mira09725123-313 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout:809: finishing write tid 4 to mira09725123-313 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 313 v 1155 (ObjNum 616 snap 29 seq_num 616) dirty exists 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout:809: done (1 left) 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:15,756.756 INFO:tasks.rados.rados.0.mira097.stdout:805: expect (ObjNum 253 snap 0 seq_num 253) 2014-08-22 19:26:16,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.084061 7f51beffd700 10 client.?.objecter ms_handle_connect 0x7f51a0000e90 2014-08-22 19:26:16,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.084072 7f51beffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:16,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.085900 7f51c673f700 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:16,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.085995 7f51c673f700 1 librados: init done 2014-08-22 19:26:16,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086381 7f51beffd700 10 client.4181.objecter ms_dispatch 0x7f51c0000f80 osd_map(100..100 src has 1..100) v3 2014-08-22 19:26:16,179.179 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086387 7f51beffd700 3 client.4181.objecter handle_osd_map got epochs [100,100] > 0 2014-08-22 19:26:16,179.179 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086389 7f51beffd700 3 client.4181.objecter handle_osd_map decoding full epoch 100 2014-08-22 19:26:16,180.180 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086456 7f51beffd700 20 client.4181.objecter dump_active .. 0 homeless 2014-08-22 19:26:16,180.180 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086493 7f51beffd700 10 client.4181.objecter ms_dispatch 0x7f51c0000f80 osd_map(100..100 src has 1..100) v3 2014-08-22 19:26:16,180.180 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086497 7f51beffd700 3 client.4181.objecter handle_osd_map ignoring epochs [100,100] <= 100 2014-08-22 19:26:16,180.180 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.086498 7f51beffd700 20 client.4181.objecter dump_active .. 0 homeless 2014-08-22 19:26:16,619.619 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:26:16,622.622 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.529842 7f51c673f700 20 client.4181.objecter shutdown clearing up homeless session... 2014-08-22 19:26:16,623.623 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:09.530237 7f51c673f700 1 librados: shutdown 2014-08-22 19:26:16,627.627 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:26:17,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.149091 7f893dc2a700 1 librados: starting msgr at :/0 2014-08-22 19:26:17,243.243 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.149099 7f893dc2a700 1 librados: starting objecter 2014-08-22 19:26:17,243.243 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.149303 7f893dc2a700 1 librados: setting wanted keys 2014-08-22 19:26:17,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.149308 7f893dc2a700 1 librados: calling monclient init 2014-08-22 19:26:17,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.150099 7f89367fc700 10 client.?.objecter ms_handle_connect 0x7f89380355e0 2014-08-22 19:26:17,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.150119 7f89367fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:17,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.151808 7f893dc2a700 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:17,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.151919 7f893dc2a700 1 librados: init done 2014-08-22 19:26:17,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152130 7f89367fc700 10 client.4184.objecter ms_dispatch 0x7f8938000f80 osd_map(101..101 src has 1..101) v3 2014-08-22 19:26:17,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152140 7f89367fc700 3 client.4184.objecter handle_osd_map got epochs [101,101] > 0 2014-08-22 19:26:17,345.345 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152143 7f89367fc700 3 client.4184.objecter handle_osd_map decoding full epoch 101 2014-08-22 19:26:17,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152212 7f89367fc700 20 client.4184.objecter dump_active .. 0 homeless 2014-08-22 19:26:17,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152259 7f89367fc700 10 client.4184.objecter ms_dispatch 0x7f8938000f80 osd_map(101..101 src has 1..101) v3 2014-08-22 19:26:17,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152266 7f89367fc700 3 client.4184.objecter handle_osd_map ignoring epochs [101,101] <= 101 2014-08-22 19:26:17,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.152269 7f89367fc700 20 client.4184.objecter dump_active .. 0 homeless 2014-08-22 19:26:17,636.636 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:26:17,639.639 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.546699 7f893dc2a700 20 client.4184.objecter shutdown clearing up homeless session... 2014-08-22 19:26:17,639.639 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:10.547153 7f893dc2a700 1 librados: shutdown 2014-08-22 19:26:17,644.644 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:26:17,658.658 INFO:tasks.rados.rados.0.mira097.stdout:805: done (0 left) 2014-08-22 19:26:18,118.118 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.025743 7f557ea12700 1 librados: starting msgr at :/0 2014-08-22 19:26:18,118.118 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.025753 7f557ea12700 1 librados: starting objecter 2014-08-22 19:26:18,118.118 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.026067 7f557ea12700 1 librados: setting wanted keys 2014-08-22 19:26:18,118.118 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.026076 7f557ea12700 1 librados: calling monclient init 2014-08-22 19:26:18,119.119 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.027725 7f556f7fe700 10 client.?.objecter ms_handle_connect 0x7f55780355e0 2014-08-22 19:26:18,119.119 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.027770 7f556f7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:18,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.030738 7f557ea12700 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:18,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.030941 7f557ea12700 1 librados: init done 2014-08-22 19:26:18,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031787 7f556f7fe700 10 client.4185.objecter ms_dispatch 0x7f5578000f80 osd_map(102..102 src has 1..102) v3 2014-08-22 19:26:18,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031801 7f556f7fe700 3 client.4185.objecter handle_osd_map got epochs [102,102] > 0 2014-08-22 19:26:18,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031805 7f556f7fe700 3 client.4185.objecter handle_osd_map decoding full epoch 102 2014-08-22 19:26:18,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031936 7f556f7fe700 20 client.4185.objecter dump_active .. 0 homeless 2014-08-22 19:26:18,124.124 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031988 7f556f7fe700 10 client.4185.objecter ms_dispatch 0x7f5578000f80 osd_map(102..102 src has 1..102) v3 2014-08-22 19:26:18,125.125 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031995 7f556f7fe700 3 client.4185.objecter handle_osd_map ignoring epochs [102,102] <= 102 2014-08-22 19:26:18,125.125 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.031998 7f556f7fe700 20 client.4185.objecter dump_active .. 0 homeless 2014-08-22 19:26:18,628.628 INFO:tasks.rados.rados.0.mira097.stdout:810: done (0 left) 2014-08-22 19:26:18,628.628 INFO:tasks.rados.rados.0.mira097.stdout:811: delete oid 299 current snap is 30 2014-08-22 19:26:18,631.631 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:26:18,636.636 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.543428 7f557ea12700 20 client.4185.objecter shutdown clearing up homeless session... 2014-08-22 19:26:18,637.637 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:11.544050 7f557ea12700 1 librados: shutdown 2014-08-22 19:26:18,645.645 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:26:18,662.662 INFO:tasks.rados.rados.0.mira097.stdout:811: done (0 left) 2014-08-22 19:26:18,662.662 INFO:tasks.rados.rados.0.mira097.stdout:812: rollback oid 2 current snap is 30 2014-08-22 19:26:18,662.662 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 2 to 28 2014-08-22 19:26:18,662.662 INFO:tasks.rados.rados.0.mira097.stdout:813: read oid 473 snap 29 2014-08-22 19:26:18,662.662 INFO:tasks.rados.rados.0.mira097.stdout:814: snap_create 2014-08-22 19:26:18,663.663 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:18,708.708 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 2 v 1087 (ObjNum 1 snap 0 seq_num 1) dirty exists 2014-08-22 19:26:18,708.708 INFO:tasks.rados.rados.0.mira097.stdout:812: done (1 left) 2014-08-22 19:26:18,800.800 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:18,800.800 INFO:tasks.rados.rados.0.mira097.stdout:813: expect (ObjNum 472 snap 0 seq_num 472) 2014-08-22 19:26:19,095.095 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.002133 7ff917fbf700 1 librados: starting msgr at :/0 2014-08-22 19:26:19,096.096 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.002139 7ff917fbf700 1 librados: starting objecter 2014-08-22 19:26:19,096.096 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.002244 7ff917fbf700 1 librados: setting wanted keys 2014-08-22 19:26:19,096.096 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.002247 7ff917fbf700 1 librados: calling monclient init 2014-08-22 19:26:19,097.097 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.004081 7ff91506c700 10 client.?.objecter ms_handle_connect 0x7ff9100355e0 2014-08-22 19:26:19,097.097 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.004141 7ff91506c700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:19,100.100 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.007801 7ff917fbf700 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:19,100.100 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.007908 7ff917fbf700 1 librados: init done 2014-08-22 19:26:19,101.101 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.008957 7ff91506c700 10 client.4198.objecter ms_dispatch 0x7ff910000f80 osd_map(103..103 src has 1..103) v3 2014-08-22 19:26:19,101.101 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.008968 7ff91506c700 3 client.4198.objecter handle_osd_map got epochs [103,103] > 0 2014-08-22 19:26:19,102.102 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.008971 7ff91506c700 3 client.4198.objecter handle_osd_map decoding full epoch 103 2014-08-22 19:26:19,102.102 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.009070 7ff91506c700 20 client.4198.objecter dump_active .. 0 homeless 2014-08-22 19:26:19,102.102 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.009174 7ff91506c700 10 client.4198.objecter ms_dispatch 0x7ff910000f80 osd_map(103..103 src has 1..103) v3 2014-08-22 19:26:19,102.102 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.009178 7ff91506c700 3 client.4198.objecter handle_osd_map ignoring epochs [103,103] <= 103 2014-08-22 19:26:19,103.103 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.009180 7ff91506c700 20 client.4198.objecter dump_active .. 0 homeless 2014-08-22 19:26:19,644.644 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:26:19,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.554586 7ff917fbf700 20 client.4198.objecter shutdown clearing up homeless session... 2014-08-22 19:26:19,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:12.554966 7ff917fbf700 1 librados: shutdown 2014-08-22 19:26:19,651.651 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:26:19,842.842 INFO:tasks.rados.rados.0.mira097.stdout:813: done (0 left) 2014-08-22 19:26:20,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.200102 7fb300d20700 1 librados: starting msgr at :/0 2014-08-22 19:26:20,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.200109 7fb300d20700 1 librados: starting objecter 2014-08-22 19:26:20,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.200401 7fb300d20700 1 librados: setting wanted keys 2014-08-22 19:26:20,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.200411 7fb300d20700 1 librados: calling monclient init 2014-08-22 19:26:20,294.294 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.201907 7fb2f9d8a700 10 client.?.objecter ms_handle_connect 0x7fb2fc0355e0 2014-08-22 19:26:20,294.294 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.202005 7fb2f9d8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:20,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.204925 7fb300d20700 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:20,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205106 7fb300d20700 1 librados: init done 2014-08-22 19:26:20,371.371 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205594 7fb2f9d8a700 10 client.4188.objecter ms_dispatch 0x7fb2fc000f80 osd_map(104..104 src has 1..104) v3 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205616 7fb2f9d8a700 3 client.4188.objecter handle_osd_map got epochs [104,104] > 0 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205624 7fb2f9d8a700 3 client.4188.objecter handle_osd_map decoding full epoch 104 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205798 7fb2f9d8a700 20 client.4188.objecter dump_active .. 0 homeless 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205853 7fb2f9d8a700 10 client.4188.objecter ms_dispatch 0x7fb2fc000f80 osd_map(104..104 src has 1..104) v3 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205858 7fb2f9d8a700 3 client.4188.objecter handle_osd_map ignoring epochs [104,104] <= 104 2014-08-22 19:26:20,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.205861 7fb2f9d8a700 20 client.4188.objecter dump_active .. 0 homeless 2014-08-22 19:26:20,647.647 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:26:20,648.648 INFO:tasks.rados.rados.0.mira097.stdout:814: done (0 left) 2014-08-22 19:26:20,648.648 INFO:tasks.rados.rados.0.mira097.stdout:815: copy_from oid 163 from oid 428 current snap is 31 2014-08-22 19:26:20,648.648 INFO:tasks.rados.rados.0.mira097.stdout:816: rollback oid 75 current snap is 31 2014-08-22 19:26:20,648.648 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 75 to 20 2014-08-22 19:26:20,648.648 INFO:tasks.rados.rados.0.mira097.stdout:817: delete oid 287 current snap is 31 2014-08-22 19:26:20,650.650 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.559043 7fb300d20700 20 client.4188.objecter shutdown clearing up homeless session... 2014-08-22 19:26:20,651.651 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:13.559473 7fb300d20700 1 librados: shutdown 2014-08-22 19:26:20,658.658 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:26:20,658.658 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:20,725.725 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 75 v 1179 (ObjNum 445 snap 0 seq_num 445) dirty exists 2014-08-22 19:26:20,725.725 INFO:tasks.rados.rados.0.mira097.stdout:816: done (2 left) 2014-08-22 19:26:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:817: done (1 left) 2014-08-22 19:26:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:818: read oid 124 snap -1 2014-08-22 19:26:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:819: copy_from oid 234 from oid 450 current snap is 31 2014-08-22 19:26:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:820: read oid 283 snap -1 2014-08-22 19:26:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:821: rollback oid 424 current snap is 31 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 424 to 28 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:822: read oid 420 snap -1 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:823: write oid 84 current snap is 31 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:823: seq_num 619 ranges {434443=631911,1572975=699506,2834442=1} 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:823: writing mira09725123-84 from 434443 to 1066354 tid 1 2014-08-22 19:26:20,824.824 INFO:tasks.rados.rados.0.mira097.stdout:823: writing mira09725123-84 from 1572975 to 2272481 tid 2 2014-08-22 19:26:20,825.825 INFO:tasks.rados.rados.0.mira097.stdout:823: writing mira09725123-84 from 2834442 to 2834443 tid 3 2014-08-22 19:26:20,825.825 INFO:tasks.rados.rados.0.mira097.stdout:824: copy_from oid 307 from oid 443 current snap is 31 2014-08-22 19:26:20,825.825 INFO:tasks.rados.rados.0.mira097.stdout:822: expect (ObjNum 419 snap 0 seq_num 419) 2014-08-22 19:26:20,968.968 INFO:tasks.rados.rados.0.mira097.stdout:822: done (7 left) 2014-08-22 19:26:20,968.968 INFO:tasks.rados.rados.0.mira097.stdout:825: read oid 5 snap 29 2014-08-22 19:26:20,968.968 INFO:tasks.rados.rados.0.mira097.stdout:815: finishing copy_from to mira09725123-163 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 163 v 765 (ObjNum 504 snap 3 seq_num 504) dirty exists 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:819: finishing copy_from to mira09725123-234 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 234 v 995 (ObjNum 567 snap 20 seq_num 567) dirty exists 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:815: finishing copy_from racing read to mira09725123-163 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:823: finishing write tid 1 to mira09725123-84 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 424 v 1093 (ObjNum 423 snap 0 seq_num 423) dirty exists 2014-08-22 19:26:20,969.969 INFO:tasks.rados.rados.0.mira097.stdout:823: finishing write tid 2 to mira09725123-84 2014-08-22 19:26:20,970.970 INFO:tasks.rados.rados.0.mira097.stdout:823: finishing write tid 3 to mira09725123-84 2014-08-22 19:26:20,970.970 INFO:tasks.rados.rados.0.mira097.stdout:823: finishing write tid 5 to mira09725123-84 2014-08-22 19:26:20,970.970 INFO:tasks.rados.rados.0.mira097.stdout:824: finishing copy_from to mira09725123-307 2014-08-22 19:26:20,970.970 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 307 v 922 (ObjNum 569 snap 21 seq_num 569) dirty exists 2014-08-22 19:26:20,970.970 INFO:tasks.rados.rados.0.mira097.stdout:820: expect (ObjNum 282 snap 0 seq_num 282) 2014-08-22 19:26:21,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.081470 7facd6821700 1 librados: starting msgr at :/0 2014-08-22 19:26:21,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.081486 7facd6821700 1 librados: starting objecter 2014-08-22 19:26:21,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.081651 7facd6821700 1 librados: setting wanted keys 2014-08-22 19:26:21,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.081656 7facd6821700 1 librados: calling monclient init 2014-08-22 19:26:21,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.082501 7facc77fe700 10 client.?.objecter ms_handle_connect 0x7facd00355e0 2014-08-22 19:26:21,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.082528 7facc77fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:21,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084097 7facd6821700 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:21,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084207 7facd6821700 1 librados: init done 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084475 7facc77fe700 10 client.4187.objecter ms_dispatch 0x7facd0000f80 osd_map(105..105 src has 1..105) v3 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084484 7facc77fe700 3 client.4187.objecter handle_osd_map got epochs [105,105] > 0 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084487 7facc77fe700 3 client.4187.objecter handle_osd_map decoding full epoch 105 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084580 7facc77fe700 20 client.4187.objecter dump_active .. 0 homeless 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084612 7facc77fe700 10 client.4187.objecter ms_dispatch 0x7facd0000f80 osd_map(105..105 src has 1..105) v3 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084615 7facc77fe700 3 client.4187.objecter handle_osd_map ignoring epochs [105,105] <= 105 2014-08-22 19:26:21,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.084616 7facc77fe700 20 client.4187.objecter dump_active .. 0 homeless 2014-08-22 19:26:21,355.355 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:21,532.532 INFO:tasks.rados.rados.0.mira097.stdout:819: finishing copy_from racing read to mira09725123-234 2014-08-22 19:26:21,533.533 INFO:tasks.rados.rados.0.mira097.stdout:818: expect (ObjNum 123 snap 0 seq_num 123) 2014-08-22 19:26:21,533.533 INFO:tasks.rados.rados.0.mira097.stdout:823: finishing write tid 6 to mira09725123-84 2014-08-22 19:26:21,533.533 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 84 v 1235 (ObjNum 619 snap 31 seq_num 619) dirty exists 2014-08-22 19:26:21,533.533 INFO:tasks.rados.rados.0.mira097.stdout:824: finishing copy_from racing read to mira09725123-307 2014-08-22 19:26:21,533.533 INFO:tasks.rados.rados.0.mira097.stdout:825: expect (ObjNum 4 snap 0 seq_num 4) 2014-08-22 19:26:21,533.533 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.171696 7facd6821700 20 client.4187.objecter shutdown clearing up homeless session... 2014-08-22 19:26:21,534.534 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:14.172110 7facd6821700 1 librados: shutdown 2014-08-22 19:26:21,706.706 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:21,870.870 INFO:tasks.rados.rados.0.mira097.stdout:815: done (7 left) 2014-08-22 19:26:21,870.870 INFO:tasks.rados.rados.0.mira097.stdout:818: done (6 left) 2014-08-22 19:26:21,870.870 INFO:tasks.rados.rados.0.mira097.stdout:819: done (5 left) 2014-08-22 19:26:21,870.870 INFO:tasks.rados.rados.0.mira097.stdout:820: done (4 left) 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:821: done (3 left) 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:823: done (2 left) 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:824: done (1 left) 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:825: done (0 left) 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:826: read oid 251 snap 23 2014-08-22 19:26:21,871.871 INFO:tasks.rados.rados.0.mira097.stdout:827: snap_remove snap 30 2014-08-22 19:26:21,894.894 INFO:tasks.rados.rados.0.mira097.stdout:826: expect (ObjNum 250 snap 0 seq_num 250) 2014-08-22 19:26:22,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.197196 7f04f284f700 1 librados: starting msgr at :/0 2014-08-22 19:26:22,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.197205 7f04f284f700 1 librados: starting objecter 2014-08-22 19:26:22,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.197350 7f04f284f700 1 librados: setting wanted keys 2014-08-22 19:26:22,291.291 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.197354 7f04f284f700 1 librados: calling monclient init 2014-08-22 19:26:22,292.292 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.198080 7f04e37fe700 10 client.?.objecter ms_handle_connect 0x7f04ec0355e0 2014-08-22 19:26:22,292.292 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.198103 7f04e37fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.199475 7f04f284f700 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.199567 7f04f284f700 1 librados: init done 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.199898 7f04e37fe700 10 client.4190.objecter ms_dispatch 0x7f04ec000f80 osd_map(105..105 src has 1..105) v3 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.199909 7f04e37fe700 3 client.4190.objecter handle_osd_map got epochs [105,105] > 0 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.199912 7f04e37fe700 3 client.4190.objecter handle_osd_map decoding full epoch 105 2014-08-22 19:26:22,293.293 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.200014 7f04e37fe700 20 client.4190.objecter dump_active .. 0 homeless 2014-08-22 19:26:22,294.294 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.200062 7f04e37fe700 10 client.4190.objecter ms_dispatch 0x7f04ec000f80 osd_map(105..105 src has 1..105) v3 2014-08-22 19:26:22,294.294 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.200069 7f04e37fe700 3 client.4190.objecter handle_osd_map ignoring epochs [105,105] <= 105 2014-08-22 19:26:22,294.294 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.200072 7f04e37fe700 20 client.4190.objecter dump_active .. 0 homeless 2014-08-22 19:26:22,372.372 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:22,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.283043 7f04f284f700 20 client.4190.objecter shutdown clearing up homeless session... 2014-08-22 19:26:22,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:15.283434 7f04f284f700 1 librados: shutdown 2014-08-22 19:26:22,659.659 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:26:22,665.665 INFO:tasks.rados.rados.0.mira097.stdout:826: done (1 left) 2014-08-22 19:26:22,665.665 INFO:tasks.rados.rados.0.mira097.stdout:827: done (0 left) 2014-08-22 19:26:22,666.666 INFO:tasks.rados.rados.0.mira097.stdout:828: rollback oid 236 current snap is 31 2014-08-22 19:26:22,666.666 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 236 to 29 2014-08-22 19:26:22,666.666 INFO:tasks.rados.rados.0.mira097.stdout:829: snap_create 2014-08-22 19:26:22,666.666 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:22,680.680 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 236 v 1191 (ObjNum 235 snap 0 seq_num 235) dirty exists 2014-08-22 19:26:22,680.680 INFO:tasks.rados.rados.0.mira097.stdout:828: done (0 left) 2014-08-22 19:26:23,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.036940 7f27cbf55700 1 librados: starting msgr at :/0 2014-08-22 19:26:23,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.036956 7f27cbf55700 1 librados: starting objecter 2014-08-22 19:26:23,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.037165 7f27cbf55700 1 librados: setting wanted keys 2014-08-22 19:26:23,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.037171 7f27cbf55700 1 librados: calling monclient init 2014-08-22 19:26:23,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.038589 7f27c9002700 10 client.?.objecter ms_handle_connect 0x7f27c40355e0 2014-08-22 19:26:23,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.038648 7f27c9002700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:23,134.134 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041032 7f27cbf55700 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:23,134.134 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041202 7f27cbf55700 1 librados: init done 2014-08-22 19:26:23,134.134 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041838 7f27c9002700 10 client.4191.objecter ms_dispatch 0x7f27c4000f80 osd_map(106..106 src has 1..106) v3 2014-08-22 19:26:23,134.134 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041845 7f27c9002700 3 client.4191.objecter handle_osd_map got epochs [106,106] > 0 2014-08-22 19:26:23,135.135 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041848 7f27c9002700 3 client.4191.objecter handle_osd_map decoding full epoch 106 2014-08-22 19:26:23,135.135 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.041983 7f27c9002700 20 client.4191.objecter dump_active .. 0 homeless 2014-08-22 19:26:23,135.135 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.042037 7f27c9002700 10 client.4191.objecter ms_dispatch 0x7f27c4000f80 osd_map(106..106 src has 1..106) v3 2014-08-22 19:26:23,136.136 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.042044 7f27c9002700 3 client.4191.objecter handle_osd_map ignoring epochs [106,106] <= 106 2014-08-22 19:26:23,136.136 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.042047 7f27c9002700 20 client.4191.objecter dump_active .. 0 homeless 2014-08-22 19:26:23,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.131183 7f27cbf55700 20 client.4191.objecter shutdown clearing up homeless session... 2014-08-22 19:26:23,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:16.132095 7f27cbf55700 1 librados: shutdown 2014-08-22 19:26:23,230.230 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003786', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001834', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:25:35.667509', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873981888, u'health': u'HEALTH_OK', u'kb_used': 22799164, u'store_stats': {u'bytes_total': 5250095, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.468804', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 877079888, u'health': u'HEALTH_OK', u'kb_used': 19701164, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.072747', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873980544, u'health': u'HEALTH_OK', u'kb_used': 22800508, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.a has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 72, u'data_bytes': 1291870653, u'bytes_used': 3482759168, u'op_per_sec': 16, u'version': 178, u'pgs_by_state': [{u'count': 72, u'state_name': u'active+clean'}], u'read_bytes_sec': 1627557, u'write_bytes_sec': 0, u'bytes_avail': 5994816286720}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 106, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:26:23,231.231 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:26:23,232.232 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:23,232.232 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-22 19:26:23,232.232 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:26:23,232.232 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:26:23,232.232 INFO:teuthology.orchestra.run.mira097:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-22 19:26:23,681.681 INFO:tasks.rados.rados.0.mira097.stdout:829: done (0 left) 2014-08-22 19:26:23,681.681 INFO:tasks.rados.rados.0.mira097.stdout:830: snap_remove snap 26 2014-08-22 19:26:24,745.745 INFO:tasks.rados.rados.0.mira097.stdout:830: done (0 left) 2014-08-22 19:26:24,746.746 INFO:tasks.rados.rados.0.mira097.stdout:831: rollback oid 406 current snap is 32 2014-08-22 19:26:24,746.746 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 406 to 20 2014-08-22 19:26:24,746.746 INFO:tasks.rados.rados.0.mira097.stdout:832: copy_from oid 480 from oid 142 current snap is 32 2014-08-22 19:26:24,746.746 INFO:tasks.rados.rados.0.mira097.stdout:833: delete oid 419 current snap is 32 2014-08-22 19:26:24,746.746 INFO:tasks.rados.rados.0.mira097.stdout:833: done (2 left) 2014-08-22 19:26:24,747.747 INFO:tasks.rados.rados.0.mira097.stdout:834: write oid 34 current snap is 32 2014-08-22 19:26:24,747.747 INFO:tasks.rados.rados.0.mira097.stdout:834: seq_num 621 ranges {473478=459168,1531415=591320,2581323=642573,3273477=1} 2014-08-22 19:26:24,766.766 INFO:tasks.rados.rados.0.mira097.stdout:834: writing mira09725123-34 from 473478 to 932646 tid 1 2014-08-22 19:26:24,800.800 INFO:tasks.rados.rados.0.mira097.stdout:834: writing mira09725123-34 from 1531415 to 2122735 tid 2 2014-08-22 19:26:24,833.833 INFO:tasks.rados.rados.0.mira097.stdout:834: writing mira09725123-34 from 2581323 to 3223896 tid 3 2014-08-22 19:26:24,835.835 INFO:tasks.rados.rados.0.mira097.stdout:834: writing mira09725123-34 from 3273477 to 3273478 tid 4 2014-08-22 19:26:24,838.838 INFO:tasks.rados.rados.0.mira097.stdout:835: read oid 270 snap -1 2014-08-22 19:26:24,838.838 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 1 to mira09725123-34 2014-08-22 19:26:24,838.838 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 2 to mira09725123-34 2014-08-22 19:26:24,839.839 INFO:tasks.rados.rados.0.mira097.stdout:836: snap_remove snap 23 2014-08-22 19:26:24,851.851 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 3 to mira09725123-34 2014-08-22 19:26:24,851.851 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 4 to mira09725123-34 2014-08-22 19:26:24,851.851 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 6 to mira09725123-34 2014-08-22 19:26:24,851.851 INFO:tasks.rados.rados.0.mira097.stdout:834: finishing write tid 7 to mira09725123-34 2014-08-22 19:26:24,852.852 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 34 v 1240 (ObjNum 621 snap 32 seq_num 621) dirty exists 2014-08-22 19:26:25,277.277 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 406 v 1195 (ObjNum 405 snap 0 seq_num 405) dirty exists 2014-08-22 19:26:25,315.315 INFO:tasks.rados.rados.0.mira097.stdout:832: finishing copy_from to mira09725123-480 2014-08-22 19:26:25,315.315 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 480 v 160 (ObjNum 141 snap 0 seq_num 141) dirty exists 2014-08-22 19:26:25,334.334 INFO:tasks.rados.rados.0.mira097.stdout:832: finishing copy_from racing read to mira09725123-480 2014-08-22 19:26:25,697.697 INFO:tasks.rados.rados.0.mira097.stdout:831: done (4 left) 2014-08-22 19:26:25,698.698 INFO:tasks.rados.rados.0.mira097.stdout:832: done (3 left) 2014-08-22 19:26:25,698.698 INFO:tasks.rados.rados.0.mira097.stdout:834: done (2 left) 2014-08-22 19:26:25,698.698 INFO:tasks.rados.rados.0.mira097.stdout:835: done (1 left) 2014-08-22 19:26:25,698.698 INFO:tasks.rados.rados.0.mira097.stdout:836: done (0 left) 2014-08-22 19:26:25,699.699 INFO:tasks.rados.rados.0.mira097.stdout:837: delete oid 3 current snap is 32 2014-08-22 19:26:25,728.728 INFO:tasks.rados.rados.0.mira097.stdout:837: done (0 left) 2014-08-22 19:26:25,729.729 INFO:tasks.rados.rados.0.mira097.stdout:838: read oid 52 snap -1 2014-08-22 19:26:25,729.729 INFO:tasks.rados.rados.0.mira097.stdout:839: copy_from oid 146 from oid 458 current snap is 32 2014-08-22 19:26:25,729.729 INFO:tasks.rados.rados.0.mira097.stdout:840: read oid 454 snap -1 2014-08-22 19:26:25,729.729 INFO:tasks.rados.rados.0.mira097.stdout:841: read oid 247 snap -1 2014-08-22 19:26:25,729.729 INFO:tasks.rados.rados.0.mira097.stdout:842: snap_remove snap 28 2014-08-22 19:26:25,770.770 INFO:tasks.rados.rados.0.mira097.stdout:838: expect (ObjNum 562 snap 20 seq_num 562) 2014-08-22 19:26:26,454.454 INFO:tasks.rados.rados.0.mira097.stdout:840: expect (ObjNum 453 snap 0 seq_num 453) 2014-08-22 19:26:26,613.613 INFO:tasks.rados.rados.0.mira097.stdout:839: finishing copy_from to mira09725123-146 2014-08-22 19:26:26,613.613 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 146 v 846 (ObjNum 457 snap 0 seq_num 457) dirty exists 2014-08-22 19:26:26,613.613 INFO:tasks.rados.rados.0.mira097.stdout:839: finishing copy_from racing read to mira09725123-146 2014-08-22 19:26:26,614.614 INFO:tasks.rados.rados.0.mira097.stdout:841: expect (ObjNum 398 snap 0 seq_num 398) 2014-08-22 19:26:27,513.513 INFO:tasks.rados.rados.0.mira097.stdout:838: done (4 left) 2014-08-22 19:26:27,513.513 INFO:tasks.rados.rados.0.mira097.stdout:839: done (3 left) 2014-08-22 19:26:27,513.513 INFO:tasks.rados.rados.0.mira097.stdout:840: done (2 left) 2014-08-22 19:26:27,513.513 INFO:tasks.rados.rados.0.mira097.stdout:841: done (1 left) 2014-08-22 19:26:27,514.514 INFO:tasks.rados.rados.0.mira097.stdout:842: done (0 left) 2014-08-22 19:26:27,514.514 INFO:tasks.rados.rados.0.mira097.stdout:843: copy_from oid 158 from oid 21 current snap is 32 2014-08-22 19:26:27,514.514 INFO:tasks.rados.rados.0.mira097.stdout:844: write oid 97 current snap is 32 2014-08-22 19:26:27,514.514 INFO:tasks.rados.rados.0.mira097.stdout:844: seq_num 623 ranges {478466=555225,1278465=1} 2014-08-22 19:26:27,539.539 INFO:tasks.rados.rados.0.mira097.stdout:844: writing mira09725123-97 from 478466 to 1033691 tid 1 2014-08-22 19:26:27,542.542 INFO:tasks.rados.rados.0.mira097.stdout:844: writing mira09725123-97 from 1278465 to 1278466 tid 2 2014-08-22 19:26:27,544.544 INFO:tasks.rados.rados.0.mira097.stdout:845: snap_remove snap 20 2014-08-22 19:26:27,602.602 INFO:tasks.rados.rados.0.mira097.stdout:844: finishing write tid 1 to mira09725123-97 2014-08-22 19:26:27,606.606 INFO:tasks.rados.rados.0.mira097.stdout:844: finishing write tid 2 to mira09725123-97 2014-08-22 19:26:27,606.606 INFO:tasks.rados.rados.0.mira097.stdout:844: finishing write tid 4 to mira09725123-97 2014-08-22 19:26:27,620.620 INFO:tasks.rados.rados.0.mira097.stdout:843: finishing copy_from to mira09725123-158 2014-08-22 19:26:27,620.620 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 158 v 22 (ObjNum 20 snap 0 seq_num 20) dirty exists 2014-08-22 19:26:27,621.621 INFO:tasks.rados.rados.0.mira097.stdout:843: finishing copy_from racing read to mira09725123-158 2014-08-22 19:26:27,624.624 INFO:tasks.rados.rados.0.mira097.stdout:844: finishing write tid 5 to mira09725123-97 2014-08-22 19:26:27,624.624 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 97 v 1235 (ObjNum 623 snap 32 seq_num 623) dirty exists 2014-08-22 19:26:27,744.744 INFO:tasks.rados.rados.0.mira097.stdout:843: done (2 left) 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:844: done (1 left) 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:845: done (0 left) 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:846: rollback oid 136 current snap is 32 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 136 to 29 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:847: read oid 50 snap -1 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:848: write oid 350 current snap is 32 2014-08-22 19:26:27,745.745 INFO:tasks.rados.rados.0.mira097.stdout:848: seq_num 624 ranges {574720=400000} 2014-08-22 19:26:27,767.767 INFO:tasks.rados.rados.0.mira097.stdout:848: writing mira09725123-350 from 574720 to 974720 tid 1 2014-08-22 19:26:27,769.769 INFO:tasks.rados.rados.0.mira097.stdout:849: read oid 383 snap -1 2014-08-22 19:26:27,769.769 INFO:tasks.rados.rados.0.mira097.stdout:850: read oid 330 snap -1 2014-08-22 19:26:27,769.769 INFO:tasks.rados.rados.0.mira097.stdout:851: write oid 131 current snap is 32 2014-08-22 19:26:27,769.769 INFO:tasks.rados.rados.0.mira097.stdout:851: seq_num 625 ranges {519549=533905,1719548=1} 2014-08-22 19:26:27,789.789 INFO:tasks.rados.rados.0.mira097.stdout:851: writing mira09725123-131 from 519549 to 1053454 tid 1 2014-08-22 19:26:27,794.794 INFO:tasks.rados.rados.0.mira097.stdout:851: writing mira09725123-131 from 1719548 to 1719549 tid 2 2014-08-22 19:26:27,795.795 INFO:tasks.rados.rados.0.mira097.stdout:852: snap_remove snap 31 2014-08-22 19:26:27,811.811 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 136 v 1281 (ObjNum 135 snap 0 seq_num 135) dirty exists 2014-08-22 19:26:27,921.921 INFO:tasks.rados.rados.0.mira097.stdout:851: finishing write tid 1 to mira09725123-131 2014-08-22 19:26:27,922.922 INFO:tasks.rados.rados.0.mira097.stdout:851: finishing write tid 2 to mira09725123-131 2014-08-22 19:26:27,922.922 INFO:tasks.rados.rados.0.mira097.stdout:851: finishing write tid 4 to mira09725123-131 2014-08-22 19:26:27,923.923 INFO:tasks.rados.rados.0.mira097.stdout:851: finishing write tid 5 to mira09725123-131 2014-08-22 19:26:27,923.923 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 131 v 1174 (ObjNum 625 snap 32 seq_num 625) dirty exists 2014-08-22 19:26:27,932.932 INFO:tasks.rados.rados.0.mira097.stdout:847: expect (ObjNum 49 snap 0 seq_num 49) 2014-08-22 19:26:29,483.483 INFO:tasks.rados.rados.0.mira097.stdout:848: finishing write tid 1 to mira09725123-350 2014-08-22 19:26:29,484.484 INFO:tasks.rados.rados.0.mira097.stdout:848: finishing write tid 3 to mira09725123-350 2014-08-22 19:26:29,484.484 INFO:tasks.rados.rados.0.mira097.stdout:848: finishing write tid 4 to mira09725123-350 2014-08-22 19:26:29,484.484 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 350 v 1288 (ObjNum 624 snap 32 seq_num 624) dirty exists 2014-08-22 19:26:29,484.484 INFO:tasks.rados.rados.0.mira097.stdout:849: expect (ObjNum 382 snap 0 seq_num 382) 2014-08-22 19:26:29,848.848 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:29,849.849 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:26:30,297.297 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.203660 7fef006a4700 1 librados: starting msgr at :/0 2014-08-22 19:26:30,298.298 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.203666 7fef006a4700 1 librados: starting objecter 2014-08-22 19:26:30,298.298 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.203791 7fef006a4700 1 librados: setting wanted keys 2014-08-22 19:26:30,298.298 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.203797 7fef006a4700 1 librados: calling monclient init 2014-08-22 19:26:30,298.298 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.204567 7feefd751700 10 client.?.objecter ms_handle_connect 0x7feef80355e0 2014-08-22 19:26:30,298.298 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.204590 7feefd751700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206057 7fef006a4700 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206127 7fef006a4700 1 librados: init done 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206458 7feefd751700 10 client.4194.objecter ms_dispatch 0x7feef8000f80 osd_map(112..112 src has 1..112) v3 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206463 7feefd751700 3 client.4194.objecter handle_osd_map got epochs [112,112] > 0 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206465 7feefd751700 3 client.4194.objecter handle_osd_map decoding full epoch 112 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206555 7feefd751700 20 client.4194.objecter dump_active .. 0 homeless 2014-08-22 19:26:30,300.300 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206586 7feefd751700 10 client.4194.objecter ms_dispatch 0x7feef8000f80 osd_map(112..112 src has 1..112) v3 2014-08-22 19:26:30,301.301 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206589 7feefd751700 3 client.4194.objecter handle_osd_map ignoring epochs [112,112] <= 112 2014-08-22 19:26:30,301.301 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.206590 7feefd751700 20 client.4194.objecter dump_active .. 0 homeless 2014-08-22 19:26:30,401.401 INFO:tasks.rados.rados.0.mira097.stdout:850: expect (ObjNum 329 snap 0 seq_num 329) 2014-08-22 19:26:30,401.401 INFO:tasks.rados.rados.0.mira097.stdout:846: done (6 left) 2014-08-22 19:26:30,401.401 INFO:tasks.rados.rados.0.mira097.stdout:847: done (5 left) 2014-08-22 19:26:30,401.401 INFO:tasks.rados.rados.0.mira097.stdout:848: done (4 left) 2014-08-22 19:26:30,401.401 INFO:tasks.rados.rados.0.mira097.stdout:849: done (3 left) 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:850: done (2 left) 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:851: done (1 left) 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:852: done (0 left) 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:853: read oid 151 snap -1 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:854: read oid 315 snap -1 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:855: read oid 239 snap -1 2014-08-22 19:26:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:856: snap_create 2014-08-22 19:26:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:26:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:853: expect (ObjNum 540 snap 10 seq_num 540) 2014-08-22 19:26:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:853: done (2 left) 2014-08-22 19:26:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:30,413.413 INFO:tasks.rados.rados.0.mira097.stdout:855: expect (ObjNum 238 snap 0 seq_num 238) 2014-08-22 19:26:30,575.575 INFO:tasks.rados.rados.0.mira097.stdout:854: expect (ObjNum 314 snap 0 seq_num 314) 2014-08-22 19:26:30,792.792 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:26:30,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.702905 7fef006a4700 20 client.4194.objecter shutdown clearing up homeless session... 2014-08-22 19:26:30,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:23.703627 7fef006a4700 1 librados: shutdown 2014-08-22 19:26:30,803.803 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:26:30,918.918 INFO:tasks.rados.rados.0.mira097.stdout:854: done (1 left) 2014-08-22 19:26:30,918.918 INFO:tasks.rados.rados.0.mira097.stdout:855: done (0 left) 2014-08-22 19:26:31,381.381 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.287052 7fa93927b700 1 librados: starting msgr at :/0 2014-08-22 19:26:31,381.381 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.287062 7fa93927b700 1 librados: starting objecter 2014-08-22 19:26:31,381.381 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.287316 7fa93927b700 1 librados: setting wanted keys 2014-08-22 19:26:31,381.381 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.287330 7fa93927b700 1 librados: calling monclient init 2014-08-22 19:26:31,382.382 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.288777 7fa9322e8700 10 client.?.objecter ms_handle_connect 0x7fa9340355e0 2014-08-22 19:26:31,382.382 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.288825 7fa9322e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:31,385.385 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.291659 7fa93927b700 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:31,385.385 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.291855 7fa93927b700 1 librados: init done 2014-08-22 19:26:31,385.385 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292327 7fa9322e8700 10 client.4193.objecter ms_dispatch 0x7fa934000f80 osd_map(113..113 src has 1..113) v3 2014-08-22 19:26:31,386.386 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292345 7fa9322e8700 3 client.4193.objecter handle_osd_map got epochs [113,113] > 0 2014-08-22 19:26:31,386.386 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292351 7fa9322e8700 3 client.4193.objecter handle_osd_map decoding full epoch 113 2014-08-22 19:26:31,386.386 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292543 7fa9322e8700 20 client.4193.objecter dump_active .. 0 homeless 2014-08-22 19:26:31,387.387 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292633 7fa9322e8700 10 client.4193.objecter ms_dispatch 0x7fa934000f80 osd_map(113..113 src has 1..113) v3 2014-08-22 19:26:31,387.387 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292646 7fa9322e8700 3 client.4193.objecter handle_osd_map ignoring epochs [113,113] <= 113 2014-08-22 19:26:31,387.387 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.292652 7fa9322e8700 20 client.4193.objecter dump_active .. 0 homeless 2014-08-22 19:26:31,769.769 INFO:tasks.rados.rados.0.mira097.stdout:856: done (0 left) 2014-08-22 19:26:31,770.770 INFO:tasks.rados.rados.0.mira097.stdout:857: write oid 394 current snap is 33 2014-08-22 19:26:31,770.770 INFO:tasks.rados.rados.0.mira097.stdout:857: seq_num 626 ranges {614779=658158,1414778=1} 2014-08-22 19:26:31,786.786 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:26:31,792.792 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.699403 7fa93927b700 20 client.4193.objecter shutdown clearing up homeless session... 2014-08-22 19:26:31,792.792 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:24.699983 7fa93927b700 1 librados: shutdown 2014-08-22 19:26:31,797.797 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:26:31,806.806 INFO:tasks.rados.rados.0.mira097.stdout:857: writing mira09725123-394 from 614779 to 1272937 tid 1 2014-08-22 19:26:31,807.807 INFO:tasks.rados.rados.0.mira097.stdout:857: writing mira09725123-394 from 1414778 to 1414779 tid 2 2014-08-22 19:26:31,807.807 INFO:tasks.rados.rados.0.mira097.stdout:858: delete oid 69 current snap is 33 2014-08-22 19:26:31,821.821 INFO:tasks.rados.rados.0.mira097.stdout:857: finishing write tid 1 to mira09725123-394 2014-08-22 19:26:31,821.821 INFO:tasks.rados.rados.0.mira097.stdout:857: finishing write tid 2 to mira09725123-394 2014-08-22 19:26:31,821.821 INFO:tasks.rados.rados.0.mira097.stdout:857: finishing write tid 4 to mira09725123-394 2014-08-22 19:26:31,877.877 INFO:tasks.rados.rados.0.mira097.stdout:857: finishing write tid 5 to mira09725123-394 2014-08-22 19:26:31,877.877 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 394 v 1303 (ObjNum 626 snap 33 seq_num 626) dirty exists 2014-08-22 19:26:31,878.878 INFO:tasks.rados.rados.0.mira097.stdout:857: done (1 left) 2014-08-22 19:26:31,879.879 INFO:tasks.rados.rados.0.mira097.stdout:858: done (0 left) 2014-08-22 19:26:31,879.879 INFO:tasks.rados.rados.0.mira097.stdout:859: delete oid 152 current snap is 33 2014-08-22 19:26:31,940.940 INFO:tasks.rados.rados.0.mira097.stdout:859: done (0 left) 2014-08-22 19:26:31,940.940 INFO:tasks.rados.rados.0.mira097.stdout:860: snap_remove snap 29 2014-08-22 19:26:32,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.171636 7f0d3f3a5700 1 librados: starting msgr at :/0 2014-08-22 19:26:32,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.171644 7f0d3f3a5700 1 librados: starting objecter 2014-08-22 19:26:32,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.171896 7f0d3f3a5700 1 librados: setting wanted keys 2014-08-22 19:26:32,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.171910 7f0d3f3a5700 1 librados: calling monclient init 2014-08-22 19:26:32,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.173669 7f0d2ffff700 10 client.?.objecter ms_handle_connect 0x7f0d380355e0 2014-08-22 19:26:32,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.173759 7f0d2ffff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.176688 7f0d3f3a5700 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.176905 7f0d3f3a5700 1 librados: init done 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177460 7f0d2ffff700 10 client.4197.objecter ms_dispatch 0x7f0d38000f80 osd_map(114..114 src has 1..114) v3 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177473 7f0d2ffff700 3 client.4197.objecter handle_osd_map got epochs [114,114] > 0 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177477 7f0d2ffff700 3 client.4197.objecter handle_osd_map decoding full epoch 114 2014-08-22 19:26:32,271.271 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177620 7f0d2ffff700 20 client.4197.objecter dump_active .. 0 homeless 2014-08-22 19:26:32,272.272 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177675 7f0d2ffff700 10 client.4197.objecter ms_dispatch 0x7f0d38000f80 osd_map(114..114 src has 1..114) v3 2014-08-22 19:26:32,272.272 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177680 7f0d2ffff700 3 client.4197.objecter handle_osd_map ignoring epochs [114,114] <= 114 2014-08-22 19:26:32,272.272 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.177683 7f0d2ffff700 20 client.4197.objecter dump_active .. 0 homeless 2014-08-22 19:26:32,776.776 INFO:tasks.rados.rados.0.mira097.stdout:860: done (0 left) 2014-08-22 19:26:32,776.776 INFO:tasks.rados.rados.0.mira097.stdout:861: snap_create 2014-08-22 19:26:32,777.777 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:26:32,781.781 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.687936 7f0d3f3a5700 20 client.4197.objecter shutdown clearing up homeless session... 2014-08-22 19:26:32,781.781 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:25.688387 7f0d3f3a5700 1 librados: shutdown 2014-08-22 19:26:32,787.787 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:26:33,419.419 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.326013 7ffdc0fed700 1 librados: starting msgr at :/0 2014-08-22 19:26:33,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.326025 7ffdc0fed700 1 librados: starting objecter 2014-08-22 19:26:33,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.326327 7ffdc0fed700 1 librados: setting wanted keys 2014-08-22 19:26:33,420.420 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.326343 7ffdc0fed700 1 librados: calling monclient init 2014-08-22 19:26:33,421.421 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.328921 7ffdb9ffb700 10 client.?.objecter ms_handle_connect 0x7ffdbc0355e0 2014-08-22 19:26:33,421.421 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.328999 7ffdb9ffb700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:33,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.332826 7ffdc0fed700 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:33,426.426 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.333047 7ffdc0fed700 1 librados: init done 2014-08-22 19:26:33,427.427 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.333949 7ffdb9ffb700 10 client.4201.objecter ms_dispatch 0x7ffdbc000f80 osd_map(115..115 src has 1..115) v3 2014-08-22 19:26:33,427.427 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.333969 7ffdb9ffb700 3 client.4201.objecter handle_osd_map got epochs [115,115] > 0 2014-08-22 19:26:33,428.428 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.333975 7ffdb9ffb700 3 client.4201.objecter handle_osd_map decoding full epoch 115 2014-08-22 19:26:33,428.428 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.334143 7ffdb9ffb700 20 client.4201.objecter dump_active .. 0 homeless 2014-08-22 19:26:33,428.428 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.334246 7ffdb9ffb700 10 client.4201.objecter ms_dispatch 0x7ffdbc000f80 osd_map(115..115 src has 1..115) v3 2014-08-22 19:26:33,502.502 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.334256 7ffdb9ffb700 3 client.4201.objecter handle_osd_map ignoring epochs [115,115] <= 115 2014-08-22 19:26:33,502.502 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.334261 7ffdb9ffb700 20 client.4201.objecter dump_active .. 0 homeless 2014-08-22 19:26:33,786.786 INFO:tasks.rados.rados.0.mira097.stdout:861: done (0 left) 2014-08-22 19:26:33,786.786 INFO:tasks.rados.rados.0.mira097.stdout:862: write oid 243 current snap is 34 2014-08-22 19:26:33,786.786 INFO:tasks.rados.rados.0.mira097.stdout:862: seq_num 629 ranges {716020=609667,1903160=690802,3090327=25693} 2014-08-22 19:26:33,822.822 INFO:tasks.rados.rados.0.mira097.stdout:862: writing mira09725123-243 from 716020 to 1325687 tid 1 2014-08-22 19:26:33,827.827 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:26:33,829.829 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.736936 7ffdc0fed700 20 client.4201.objecter shutdown clearing up homeless session... 2014-08-22 19:26:33,829.829 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:26.737391 7ffdc0fed700 1 librados: shutdown 2014-08-22 19:26:33,834.834 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:26:33,846.846 INFO:tasks.rados.rados.0.mira097.stdout:862: writing mira09725123-243 from 1903160 to 2593962 tid 2 2014-08-22 19:26:33,853.853 INFO:tasks.rados.rados.0.mira097.stdout:862: writing mira09725123-243 from 3090327 to 3116020 tid 3 2014-08-22 19:26:33,853.853 INFO:tasks.rados.rados.0.mira097.stdout:863: read oid 374 snap -1 2014-08-22 19:26:33,854.854 INFO:tasks.rados.rados.0.mira097.stdout:862: finishing write tid 1 to mira09725123-243 2014-08-22 19:26:33,854.854 INFO:tasks.rados.rados.0.mira097.stdout:864: write oid 20 current snap is 34 2014-08-22 19:26:33,854.854 INFO:tasks.rados.rados.0.mira097.stdout:864: seq_num 630 ranges {659826=732895,2015244=609573,3386656=473170} 2014-08-22 19:26:33,878.878 INFO:tasks.rados.rados.0.mira097.stdout:864: writing mira09725123-20 from 659826 to 1392721 tid 1 2014-08-22 19:26:33,901.901 INFO:tasks.rados.rados.0.mira097.stdout:864: writing mira09725123-20 from 2015244 to 2624817 tid 2 2014-08-22 19:26:33,919.919 INFO:tasks.rados.rados.0.mira097.stdout:864: writing mira09725123-20 from 3386656 to 3859826 tid 3 2014-08-22 19:26:33,958.958 INFO:tasks.rados.rados.0.mira097.stdout:865: snap_create 2014-08-22 19:26:33,958.958 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:26:33,958.958 INFO:tasks.rados.rados.0.mira097.stdout:863: expect (ObjNum 373 snap 0 seq_num 373) 2014-08-22 19:26:34,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.278506 7f492afcb700 1 librados: starting msgr at :/0 2014-08-22 19:26:34,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.278520 7f492afcb700 1 librados: starting objecter 2014-08-22 19:26:34,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.278669 7f492afcb700 1 librados: setting wanted keys 2014-08-22 19:26:34,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.278672 7f492afcb700 1 librados: calling monclient init 2014-08-22 19:26:34,373.373 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.280396 7f4923fff700 10 client.?.objecter ms_handle_connect 0x7f49240355e0 2014-08-22 19:26:34,373.373 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.280419 7f4923fff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:34,376.376 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.283731 7f492afcb700 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:34,377.377 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.283838 7f492afcb700 1 librados: init done 2014-08-22 19:26:34,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284621 7f4923fff700 10 client.4204.objecter ms_dispatch 0x7f4924000f80 osd_map(116..116 src has 1..116) v3 2014-08-22 19:26:34,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284628 7f4923fff700 3 client.4204.objecter handle_osd_map got epochs [116,116] > 0 2014-08-22 19:26:34,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284630 7f4923fff700 3 client.4204.objecter handle_osd_map decoding full epoch 116 2014-08-22 19:26:34,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284696 7f4923fff700 20 client.4204.objecter dump_active .. 0 homeless 2014-08-22 19:26:34,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284830 7f4923fff700 10 client.4204.objecter ms_dispatch 0x7f4924000f80 osd_map(116..116 src has 1..116) v3 2014-08-22 19:26:34,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284837 7f4923fff700 3 client.4204.objecter handle_osd_map ignoring epochs [116,116] <= 116 2014-08-22 19:26:34,380.380 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.284841 7f4923fff700 20 client.4204.objecter dump_active .. 0 homeless 2014-08-22 19:26:34,818.818 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:26:34,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.728743 7f492afcb700 20 client.4204.objecter shutdown clearing up homeless session... 2014-08-22 19:26:34,822.822 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:27.729170 7f492afcb700 1 librados: shutdown 2014-08-22 19:26:34,826.826 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:26:34,865.865 INFO:tasks.rados.rados.0.mira097.stdout:862: finishing write tid 2 to mira09725123-243 2014-08-22 19:26:34,865.865 INFO:tasks.rados.rados.0.mira097.stdout:862: finishing write tid 3 to mira09725123-243 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:862: finishing write tid 5 to mira09725123-243 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:862: finishing write tid 6 to mira09725123-243 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 243 v 1270 (ObjNum 629 snap 34 seq_num 629) dirty exists 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:864: finishing write tid 1 to mira09725123-20 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:864: finishing write tid 2 to mira09725123-20 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:864: finishing write tid 3 to mira09725123-20 2014-08-22 19:26:34,866.866 INFO:tasks.rados.rados.0.mira097.stdout:864: finishing write tid 5 to mira09725123-20 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout:862: done (2 left) 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout:863: done (1 left) 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout:864: finishing write tid 6 to mira09725123-20 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 20 v 1317 (ObjNum 630 snap 34 seq_num 630) dirty exists 2014-08-22 19:26:34,867.867 INFO:tasks.rados.rados.0.mira097.stdout:864: done (0 left) 2014-08-22 19:26:35,315.315 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.221806 7f368fb7c700 1 librados: starting msgr at :/0 2014-08-22 19:26:35,316.316 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.221813 7f368fb7c700 1 librados: starting objecter 2014-08-22 19:26:35,316.316 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.221998 7f368fb7c700 1 librados: setting wanted keys 2014-08-22 19:26:35,316.316 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.222003 7f368fb7c700 1 librados: calling monclient init 2014-08-22 19:26:35,317.317 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.223446 7f368cc29700 10 client.?.objecter ms_handle_connect 0x7f36880355e0 2014-08-22 19:26:35,317.317 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.223481 7f368cc29700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:35,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.226418 7f368fb7c700 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:35,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.226565 7f368fb7c700 1 librados: init done 2014-08-22 19:26:35,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227082 7f368cc29700 10 client.4200.objecter ms_dispatch 0x7f3688000f80 osd_map(117..117 src has 1..117) v3 2014-08-22 19:26:35,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227097 7f368cc29700 3 client.4200.objecter handle_osd_map got epochs [117,117] > 0 2014-08-22 19:26:35,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227102 7f368cc29700 3 client.4200.objecter handle_osd_map decoding full epoch 117 2014-08-22 19:26:35,321.321 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227291 7f368cc29700 20 client.4200.objecter dump_active .. 0 homeless 2014-08-22 19:26:35,392.392 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227368 7f368cc29700 10 client.4200.objecter ms_dispatch 0x7f3688000f80 osd_map(117..117 src has 1..117) v3 2014-08-22 19:26:35,392.392 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227382 7f368cc29700 3 client.4200.objecter handle_osd_map ignoring epochs [117,117] <= 117 2014-08-22 19:26:35,392.392 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.227388 7f368cc29700 20 client.4200.objecter dump_active .. 0 homeless 2014-08-22 19:26:35,817.817 INFO:tasks.rados.rados.0.mira097.stdout:865: done (0 left) 2014-08-22 19:26:35,817.817 INFO:tasks.rados.rados.0.mira097.stdout:866: snap_create 2014-08-22 19:26:35,819.819 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:26:35,825.825 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.731793 7f368fb7c700 20 client.4200.objecter shutdown clearing up homeless session... 2014-08-22 19:26:35,825.825 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:28.732453 7f368fb7c700 1 librados: shutdown 2014-08-22 19:26:35,833.833 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:26:35,833.833 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:36,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.382531 7fc9ce424700 1 librados: starting msgr at :/0 2014-08-22 19:26:36,478.478 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.382550 7fc9ce424700 1 librados: starting objecter 2014-08-22 19:26:36,478.478 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.382891 7fc9ce424700 1 librados: setting wanted keys 2014-08-22 19:26:36,478.478 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.382902 7fc9ce424700 1 librados: calling monclient init 2014-08-22 19:26:36,478.478 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.385301 7fc9c6ffd700 10 client.?.objecter ms_handle_connect 0x7fc9c80355e0 2014-08-22 19:26:36,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.385361 7fc9c6ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:36,482.482 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.388842 7fc9ce424700 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:36,482.482 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.389026 7fc9ce424700 1 librados: init done 2014-08-22 19:26:36,483.483 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.389858 7fc9c6ffd700 10 client.4207.objecter ms_dispatch 0x7fc9c8000f80 osd_map(118..118 src has 1..118) v3 2014-08-22 19:26:36,483.483 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.389876 7fc9c6ffd700 3 client.4207.objecter handle_osd_map got epochs [118,118] > 0 2014-08-22 19:26:36,483.483 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.389882 7fc9c6ffd700 3 client.4207.objecter handle_osd_map decoding full epoch 118 2014-08-22 19:26:36,483.483 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.390049 7fc9c6ffd700 20 client.4207.objecter dump_active .. 0 homeless 2014-08-22 19:26:36,549.549 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.390139 7fc9c6ffd700 10 client.4207.objecter ms_dispatch 0x7fc9c8000f80 osd_map(118..118 src has 1..118) v3 2014-08-22 19:26:36,549.549 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.390152 7fc9c6ffd700 3 client.4207.objecter handle_osd_map ignoring epochs [118,118] <= 118 2014-08-22 19:26:36,549.549 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.390159 7fc9c6ffd700 20 client.4207.objecter dump_active .. 0 homeless 2014-08-22 19:26:36,661.661 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:36,896.896 INFO:tasks.rados.rados.0.mira097.stdout:866: done (0 left) 2014-08-22 19:26:36,896.896 INFO:tasks.rados.rados.0.mira097.stdout:867: write oid 64 current snap is 36 2014-08-22 19:26:36,897.897 INFO:tasks.rados.rados.0.mira097.stdout:867: seq_num 631 ranges {603631=1} 2014-08-22 19:26:36,897.897 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.491985 7fc9ce424700 20 client.4207.objecter shutdown clearing up homeless session... 2014-08-22 19:26:36,897.897 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:29.492762 7fc9ce424700 1 librados: shutdown 2014-08-22 19:26:36,907.907 INFO:tasks.rados.rados.0.mira097.stdout:867: writing mira09725123-64 from 603631 to 603632 tid 1 2014-08-22 19:26:36,996.996 INFO:tasks.rados.rados.0.mira097.stdout:868: snap_remove snap 34 2014-08-22 19:26:36,996.996 INFO:tasks.rados.rados.0.mira097.stdout:867: finishing write tid 1 to mira09725123-64 2014-08-22 19:26:36,996.996 INFO:tasks.rados.rados.0.mira097.stdout:867: finishing write tid 3 to mira09725123-64 2014-08-22 19:26:36,996.996 INFO:tasks.rados.rados.0.mira097.stdout:867: finishing write tid 4 to mira09725123-64 2014-08-22 19:26:36,997.997 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 64 v 1321 (ObjNum 631 snap 36 seq_num 631) dirty exists 2014-08-22 19:26:37,027.027 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:37,636.636 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.542472 7f67ba7fc700 1 librados: starting msgr at :/0 2014-08-22 19:26:37,636.636 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.542483 7f67ba7fc700 1 librados: starting objecter 2014-08-22 19:26:37,636.636 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.542734 7f67ba7fc700 1 librados: setting wanted keys 2014-08-22 19:26:37,636.636 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.542745 7f67ba7fc700 1 librados: calling monclient init 2014-08-22 19:26:37,637.637 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.544241 7f67ab7fe700 10 client.?.objecter ms_handle_connect 0x7f67b40355e0 2014-08-22 19:26:37,637.637 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.544303 7f67ab7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:37,641.641 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.547436 7f67ba7fc700 10 client.4203.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:37,641.641 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.547633 7f67ba7fc700 1 librados: init done 2014-08-22 19:26:37,641.641 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548124 7f67ab7fe700 10 client.4203.objecter ms_dispatch 0x7f67b4000f80 osd_map(119..119 src has 1..119) v3 2014-08-22 19:26:37,642.642 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548136 7f67ab7fe700 3 client.4203.objecter handle_osd_map got epochs [119,119] > 0 2014-08-22 19:26:37,642.642 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548141 7f67ab7fe700 3 client.4203.objecter handle_osd_map decoding full epoch 119 2014-08-22 19:26:37,642.642 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548299 7f67ab7fe700 20 client.4203.objecter dump_active .. 0 homeless 2014-08-22 19:26:37,642.642 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548395 7f67ab7fe700 10 client.4203.objecter ms_dispatch 0x7f67b4000f80 osd_map(119..119 src has 1..119) v3 2014-08-22 19:26:37,642.642 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548410 7f67ab7fe700 3 client.4203.objecter handle_osd_map ignoring epochs [119,119] <= 119 2014-08-22 19:26:37,643.643 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.548416 7f67ab7fe700 20 client.4203.objecter dump_active .. 0 homeless 2014-08-22 19:26:37,745.745 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:37,916.916 INFO:tasks.rados.rados.0.mira097.stdout:867: done (1 left) 2014-08-22 19:26:37,916.916 INFO:tasks.rados.rados.0.mira097.stdout:868: done (0 left) 2014-08-22 19:26:37,916.916 INFO:tasks.rados.rados.0.mira097.stdout:869: snap_create 2014-08-22 19:26:37,920.920 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.645352 7f67ba7fc700 20 client.4203.objecter shutdown clearing up homeless session... 2014-08-22 19:26:37,920.920 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:30.645980 7f67ba7fc700 1 librados: shutdown 2014-08-22 19:26:38,019.019 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:26:38,625.625 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.532191 7f4e96377700 1 librados: starting msgr at :/0 2014-08-22 19:26:38,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.532197 7f4e96377700 1 librados: starting objecter 2014-08-22 19:26:38,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.532432 7f4e96377700 1 librados: setting wanted keys 2014-08-22 19:26:38,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.532446 7f4e96377700 1 librados: calling monclient init 2014-08-22 19:26:38,627.627 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.534815 7f4e8effd700 10 client.?.objecter ms_handle_connect 0x7f4e900355e0 2014-08-22 19:26:38,628.628 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.534892 7f4e8effd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:38,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.538905 7f4e96377700 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:38,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.539130 7f4e96377700 1 librados: init done 2014-08-22 19:26:38,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.539898 7f4e8effd700 10 client.4210.objecter ms_dispatch 0x7f4e90000f80 osd_map(120..120 src has 1..120) v3 2014-08-22 19:26:38,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.539915 7f4e8effd700 3 client.4210.objecter handle_osd_map got epochs [120,120] > 0 2014-08-22 19:26:38,634.634 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.539920 7f4e8effd700 3 client.4210.objecter handle_osd_map decoding full epoch 120 2014-08-22 19:26:38,634.634 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.540050 7f4e8effd700 20 client.4210.objecter dump_active .. 0 homeless 2014-08-22 19:26:38,634.634 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.540098 7f4e8effd700 10 client.4210.objecter ms_dispatch 0x7f4e90000f80 osd_map(120..120 src has 1..120) v3 2014-08-22 19:26:38,634.634 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.540101 7f4e8effd700 3 client.4210.objecter handle_osd_map ignoring epochs [120,120] <= 120 2014-08-22 19:26:38,635.635 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.540103 7f4e8effd700 20 client.4210.objecter dump_active .. 0 homeless 2014-08-22 19:26:38,728.728 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.635628 7f4e96377700 20 client.4210.objecter shutdown clearing up homeless session... 2014-08-22 19:26:38,729.729 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:31.636391 7f4e96377700 1 librados: shutdown 2014-08-22 19:26:38,738.738 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003786', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001834', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:25:35.667509', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873981888, u'health': u'HEALTH_OK', u'kb_used': 22799164, u'store_stats': {u'bytes_total': 5250095, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.468804', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 877079888, u'health': u'HEALTH_OK', u'kb_used': 19701164, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:25:36.072747', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873980544, u'health': u'HEALTH_OK', u'kb_used': 22800508, u'store_stats': {u'bytes_total': 7347247, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.b has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 72, u'data_bytes': 1209058892, u'bytes_used': 3389321216, u'version': 198, u'pgs_by_state': [{u'count': 72, u'state_name': u'active+clean'}], u'bytes_avail': 5994909724672}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 120, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:26:38,740.740 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:26:38,740.740 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:38,740.740 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.663344 2014-08-22 19:26:38,741.741 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.663344443499' 2014-08-22 19:26:38,844.844 INFO:tasks.rados.rados.0.mira097.stdout:869: done (0 left) 2014-08-22 19:26:38,845.845 INFO:tasks.rados.rados.0.mira097.stdout:870: delete oid 66 current snap is 37 2014-08-22 19:26:38,864.864 INFO:tasks.rados.rados.0.mira097.stdout:870: done (0 left) 2014-08-22 19:26:38,864.864 INFO:tasks.rados.rados.0.mira097.stdout:871: write oid 497 current snap is 37 2014-08-22 19:26:38,865.865 INFO:tasks.rados.rados.0.mira097.stdout:871: seq_num 633 ranges {608619=532980,1867272=454206,2608618=1} 2014-08-22 19:26:38,894.894 INFO:tasks.rados.rados.0.mira097.stdout:871: writing mira09725123-497 from 608619 to 1141599 tid 1 2014-08-22 19:26:38,911.911 INFO:tasks.rados.rados.0.mira097.stdout:871: writing mira09725123-497 from 1867272 to 2321478 tid 2 2014-08-22 19:26:38,916.916 INFO:tasks.rados.rados.0.mira097.stdout:871: writing mira09725123-497 from 2608618 to 2608619 tid 3 2014-08-22 19:26:38,916.916 INFO:tasks.rados.rados.0.mira097.stdout:872: rollback oid 315 current snap is 37 2014-08-22 19:26:38,916.916 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 315 to 33 2014-08-22 19:26:38,917.917 INFO:tasks.rados.rados.0.mira097.stdout:871: finishing write tid 1 to mira09725123-497 2014-08-22 19:26:38,917.917 INFO:tasks.rados.rados.0.mira097.stdout:873: snap_remove snap 35 2014-08-22 19:26:38,933.933 INFO:tasks.rados.rados.0.mira097.stdout:871: finishing write tid 2 to mira09725123-497 2014-08-22 19:26:38,935.935 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 315 v 1328 (ObjNum 314 snap 0 seq_num 314) dirty exists 2014-08-22 19:26:38,936.936 INFO:tasks.rados.rados.0.mira097.stdout:871: finishing write tid 3 to mira09725123-497 2014-08-22 19:26:38,939.939 INFO:tasks.rados.rados.0.mira097.stdout:871: finishing write tid 5 to mira09725123-497 2014-08-22 19:26:38,946.946 INFO:tasks.rados.rados.0.mira097.stdout:871: finishing write tid 6 to mira09725123-497 2014-08-22 19:26:38,947.947 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 497 v 1275 (ObjNum 633 snap 37 seq_num 633) dirty exists 2014-08-22 19:26:39,317.317 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.224037 7fb48502d700 1 librados: starting msgr at :/0 2014-08-22 19:26:39,318.318 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.224044 7fb48502d700 1 librados: starting objecter 2014-08-22 19:26:39,318.318 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.224365 7fb48502d700 1 librados: setting wanted keys 2014-08-22 19:26:39,318.318 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.224379 7fb48502d700 1 librados: calling monclient init 2014-08-22 19:26:39,319.319 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.226179 7fb47e0ae700 10 client.?.objecter ms_handle_connect 0x7fb4800355e0 2014-08-22 19:26:39,319.319 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.226221 7fb47e0ae700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:39,322.322 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229012 7fb48502d700 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:39,322.322 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229183 7fb48502d700 1 librados: init done 2014-08-22 19:26:39,322.322 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229677 7fb47e0ae700 10 client.4206.objecter ms_dispatch 0x7fb480000f80 osd_map(121..121 src has 1..121) v3 2014-08-22 19:26:39,323.323 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229692 7fb47e0ae700 3 client.4206.objecter handle_osd_map got epochs [121,121] > 0 2014-08-22 19:26:39,323.323 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229695 7fb47e0ae700 3 client.4206.objecter handle_osd_map decoding full epoch 121 2014-08-22 19:26:39,323.323 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229829 7fb47e0ae700 20 client.4206.objecter dump_active .. 0 homeless 2014-08-22 19:26:39,323.323 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229959 7fb47e0ae700 10 client.4206.objecter ms_dispatch 0x7fb480000f80 osd_map(121..121 src has 1..121) v3 2014-08-22 19:26:39,324.324 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229967 7fb47e0ae700 3 client.4206.objecter handle_osd_map ignoring epochs [121,121] <= 121 2014-08-22 19:26:39,324.324 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.229970 7fb47e0ae700 20 client.4206.objecter dump_active .. 0 homeless 2014-08-22 19:26:39,864.864 INFO:tasks.rados.rados.0.mira097.stdout:871: done (2 left) 2014-08-22 19:26:39,864.864 INFO:tasks.rados.rados.0.mira097.stdout:872: done (1 left) 2014-08-22 19:26:39,864.864 INFO:tasks.rados.rados.0.mira097.stdout:873: done (0 left) 2014-08-22 19:26:39,864.864 INFO:tasks.rados.rados.0.mira097.stdout:874: read oid 487 snap -1 2014-08-22 19:26:39,864.864 INFO:tasks.rados.rados.0.mira097.stdout:875: snap_create 2014-08-22 19:26:39,865.865 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:39,879.879 INFO:teuthology.orchestra.run.mira097.stderr:set osd.4 primary-affinity to 0.663344 (8434722) 2014-08-22 19:26:39,884.884 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.790785 7fb48502d700 20 client.4206.objecter shutdown clearing up homeless session... 2014-08-22 19:26:39,884.884 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:32.791058 7fb48502d700 1 librados: shutdown 2014-08-22 19:26:39,898.898 INFO:tasks.rados.rados.0.mira097.stdout:874: expect (ObjNum 486 snap 0 seq_num 486) 2014-08-22 19:26:40,230.230 INFO:tasks.rados.rados.0.mira097.stdout:874: done (0 left) 2014-08-22 19:26:40,873.873 INFO:tasks.rados.rados.0.mira097.stdout:875: done (0 left) 2014-08-22 19:26:40,873.873 INFO:tasks.rados.rados.0.mira097.stdout:876: read oid 21 snap 37 2014-08-22 19:26:40,873.873 INFO:tasks.rados.rados.0.mira097.stdout:877: write oid 377 current snap is 38 2014-08-22 19:26:40,873.873 INFO:tasks.rados.rados.0.mira097.stdout:877: seq_num 634 ranges {704873=656208,1849180=455693} 2014-08-22 19:26:40,900.900 INFO:tasks.rados.rados.0.mira097.stdout:877: writing mira09725123-377 from 704873 to 1361081 tid 1 2014-08-22 19:26:40,921.921 INFO:tasks.rados.rados.0.mira097.stdout:877: writing mira09725123-377 from 1849180 to 2304873 tid 2 2014-08-22 19:26:40,927.927 INFO:tasks.rados.rados.0.mira097.stdout:878: snap_create 2014-08-22 19:26:40,928.928 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:40,928.928 INFO:tasks.rados.rados.0.mira097.stdout:876: expect (ObjNum 20 snap 0 seq_num 20) 2014-08-22 19:26:41,304.304 INFO:tasks.rados.rados.0.mira097.stdout:877: finishing write tid 1 to mira09725123-377 2014-08-22 19:26:41,304.304 INFO:tasks.rados.rados.0.mira097.stdout:877: finishing write tid 2 to mira09725123-377 2014-08-22 19:26:41,304.304 INFO:tasks.rados.rados.0.mira097.stdout:877: finishing write tid 4 to mira09725123-377 2014-08-22 19:26:41,305.305 INFO:tasks.rados.rados.0.mira097.stdout:877: finishing write tid 5 to mira09725123-377 2014-08-22 19:26:41,305.305 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 377 v 1282 (ObjNum 634 snap 38 seq_num 634) dirty exists 2014-08-22 19:26:41,305.305 INFO:tasks.rados.rados.0.mira097.stdout:876: done (1 left) 2014-08-22 19:26:41,305.305 INFO:tasks.rados.rados.0.mira097.stdout:877: done (0 left) 2014-08-22 19:26:41,891.891 INFO:tasks.rados.rados.0.mira097.stdout:878: done (0 left) 2014-08-22 19:26:41,891.891 INFO:tasks.rados.rados.0.mira097.stdout:879: write oid 161 current snap is 39 2014-08-22 19:26:41,891.891 INFO:tasks.rados.rados.0.mira097.stdout:879: seq_num 635 ranges {401127=628013,1811890=189237} 2014-08-22 19:26:41,916.916 INFO:tasks.rados.rados.0.mira097.stdout:879: writing mira09725123-161 from 401127 to 1029140 tid 1 2014-08-22 19:26:41,928.928 INFO:tasks.rados.rados.0.mira097.stdout:879: writing mira09725123-161 from 1811890 to 2001127 tid 2 2014-08-22 19:26:41,931.931 INFO:tasks.rados.rados.0.mira097.stdout:880: copy_from oid 445 from oid 360 current snap is 39 2014-08-22 19:26:41,931.931 INFO:tasks.rados.rados.0.mira097.stdout:881: delete oid 6 current snap is 39 2014-08-22 19:26:41,975.975 INFO:tasks.rados.rados.0.mira097.stdout:879: finishing write tid 1 to mira09725123-161 2014-08-22 19:26:41,978.978 INFO:tasks.rados.rados.0.mira097.stdout:879: finishing write tid 2 to mira09725123-161 2014-08-22 19:26:41,979.979 INFO:tasks.rados.rados.0.mira097.stdout:879: finishing write tid 4 to mira09725123-161 2014-08-22 19:26:42,112.112 INFO:tasks.rados.rados.0.mira097.stdout:880: finishing copy_from to mira09725123-445 2014-08-22 19:26:42,112.112 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 445 v 525 (ObjNum 359 snap 0 seq_num 359) dirty exists 2014-08-22 19:26:42,116.116 INFO:tasks.rados.rados.0.mira097.stdout:879: finishing write tid 5 to mira09725123-161 2014-08-22 19:26:42,117.117 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 161 v 1292 (ObjNum 635 snap 39 seq_num 635) dirty exists 2014-08-22 19:26:42,118.118 INFO:tasks.rados.rados.0.mira097.stdout:879: done (2 left) 2014-08-22 19:26:42,118.118 INFO:tasks.rados.rados.0.mira097.stdout:881: done (1 left) 2014-08-22 19:26:42,118.118 INFO:tasks.rados.rados.0.mira097.stdout:882: write oid 213 current snap is 39 2014-08-22 19:26:42,119.119 INFO:tasks.rados.rados.0.mira097.stdout:882: seq_num 637 ranges {6113=1} 2014-08-22 19:26:42,119.119 INFO:tasks.rados.rados.0.mira097.stdout:882: writing mira09725123-213 from 6113 to 6114 tid 1 2014-08-22 19:26:42,119.119 INFO:tasks.rados.rados.0.mira097.stdout:883: snap_remove snap 37 2014-08-22 19:26:42,127.127 INFO:tasks.rados.rados.0.mira097.stdout:882: finishing write tid 1 to mira09725123-213 2014-08-22 19:26:42,130.130 INFO:tasks.rados.rados.0.mira097.stdout:882: finishing write tid 3 to mira09725123-213 2014-08-22 19:26:42,136.136 INFO:tasks.rados.rados.0.mira097.stdout:880: finishing copy_from racing read to mira09725123-445 2014-08-22 19:26:42,147.147 INFO:tasks.rados.rados.0.mira097.stdout:882: finishing write tid 4 to mira09725123-213 2014-08-22 19:26:42,148.148 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 213 v 1350 (ObjNum 637 snap 39 seq_num 637) dirty exists 2014-08-22 19:26:42,924.924 INFO:tasks.rados.rados.0.mira097.stdout:880: done (2 left) 2014-08-22 19:26:42,925.925 INFO:tasks.rados.rados.0.mira097.stdout:882: done (1 left) 2014-08-22 19:26:42,925.925 INFO:tasks.rados.rados.0.mira097.stdout:883: done (0 left) 2014-08-22 19:26:42,925.925 INFO:tasks.rados.rados.0.mira097.stdout:884: write oid 138 current snap is 39 2014-08-22 19:26:42,925.925 INFO:tasks.rados.rados.0.mira097.stdout:884: seq_num 638 ranges {500320=730946,1716142=708030,3223787=476533} 2014-08-22 19:26:42,952.952 INFO:tasks.rados.rados.0.mira097.stdout:884: writing mira09725123-138 from 500320 to 1231266 tid 1 2014-08-22 19:26:42,976.976 INFO:tasks.rados.rados.0.mira097.stdout:884: writing mira09725123-138 from 1716142 to 2424172 tid 2 2014-08-22 19:26:42,994.994 INFO:tasks.rados.rados.0.mira097.stdout:884: writing mira09725123-138 from 3223787 to 3700320 tid 3 2014-08-22 19:26:42,998.998 INFO:tasks.rados.rados.0.mira097.stdout:885: delete oid 288 current snap is 39 2014-08-22 19:26:42,999.999 INFO:tasks.rados.rados.0.mira097.stdout:884: finishing write tid 1 to mira09725123-138 2014-08-22 19:26:42,999.999 INFO:tasks.rados.rados.0.mira097.stdout:884: finishing write tid 2 to mira09725123-138 2014-08-22 19:26:43,009.009 INFO:tasks.rados.rados.0.mira097.stdout:884: finishing write tid 3 to mira09725123-138 2014-08-22 19:26:43,009.009 INFO:tasks.rados.rados.0.mira097.stdout:884: finishing write tid 5 to mira09725123-138 2014-08-22 19:26:43,010.010 INFO:tasks.rados.rados.0.mira097.stdout:885: done (1 left) 2014-08-22 19:26:43,010.010 INFO:tasks.rados.rados.0.mira097.stdout:886: read oid 28 snap -1 2014-08-22 19:26:43,010.010 INFO:tasks.rados.rados.0.mira097.stdout:887: read oid 145 snap -1 2014-08-22 19:26:43,011.011 INFO:tasks.rados.rados.0.mira097.stdout:888: read oid 165 snap 38 2014-08-22 19:26:43,098.098 INFO:tasks.rados.rados.0.mira097.stdout:889: read oid 134 snap 32 2014-08-22 19:26:43,098.098 INFO:tasks.rados.rados.0.mira097.stdout:890: snap_create 2014-08-22 19:26:43,098.098 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:26:43,098.098 INFO:tasks.rados.rados.0.mira097.stdout:887: done (4 left) 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout:884: finishing write tid 6 to mira09725123-138 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 138 v 1305 (ObjNum 638 snap 39 seq_num 638) dirty exists 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout:884: done (3 left) 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:26:43,099.099 INFO:tasks.rados.rados.0.mira097.stdout:888: expect (ObjNum 534 snap 8 seq_num 534) 2014-08-22 19:26:43,433.433 INFO:tasks.rados.rados.0.mira097.stdout:886: expect (ObjNum 27 snap 0 seq_num 27) 2014-08-22 19:26:43,806.806 INFO:tasks.rados.rados.0.mira097.stdout:889: expect (ObjNum 133 snap 0 seq_num 133) 2014-08-22 19:26:43,851.851 INFO:tasks.rados.rados.0.mira097.stdout:886: done (2 left) 2014-08-22 19:26:43,851.851 INFO:tasks.rados.rados.0.mira097.stdout:888: done (1 left) 2014-08-22 19:26:43,851.851 INFO:tasks.rados.rados.0.mira097.stdout:889: done (0 left) 2014-08-22 19:26:43,930.930 INFO:tasks.rados.rados.0.mira097.stdout:890: done (0 left) 2014-08-22 19:26:43,931.931 INFO:tasks.rados.rados.0.mira097.stdout:891: snap_create 2014-08-22 19:26:44,889.889 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:44,889.889 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:44,890.890 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.608507612917 2014-08-22 19:26:44,890.890 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 0.608507612917' 2014-08-22 19:26:44,934.934 INFO:tasks.rados.rados.0.mira097.stdout:891: done (0 left) 2014-08-22 19:26:44,934.934 INFO:tasks.rados.rados.0.mira097.stdout:892: snap_create 2014-08-22 19:26:45,362.362 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.268857 7f25df89c700 1 librados: starting msgr at :/0 2014-08-22 19:26:45,363.363 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.268866 7f25df89c700 1 librados: starting objecter 2014-08-22 19:26:45,363.363 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.269028 7f25df89c700 1 librados: setting wanted keys 2014-08-22 19:26:45,363.363 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.269037 7f25df89c700 1 librados: calling monclient init 2014-08-22 19:26:45,364.364 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.270635 7f25dc949700 10 client.?.objecter ms_handle_connect 0x7f25d80355e0 2014-08-22 19:26:45,364.364 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.270713 7f25dc949700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:45,367.367 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.273640 7f25df89c700 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:45,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.273762 7f25df89c700 1 librados: init done 2014-08-22 19:26:45,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274264 7f25dc949700 10 client.4209.objecter ms_dispatch 0x7f25d8000f80 osd_map(127..127 src has 1..127) v3 2014-08-22 19:26:45,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274276 7f25dc949700 3 client.4209.objecter handle_osd_map got epochs [127,127] > 0 2014-08-22 19:26:45,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274279 7f25dc949700 3 client.4209.objecter handle_osd_map decoding full epoch 127 2014-08-22 19:26:45,368.368 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274460 7f25dc949700 20 client.4209.objecter dump_active .. 0 homeless 2014-08-22 19:26:45,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274525 7f25dc949700 10 client.4209.objecter ms_dispatch 0x7f25d8000f80 osd_map(127..127 src has 1..127) v3 2014-08-22 19:26:45,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274533 7f25dc949700 3 client.4209.objecter handle_osd_map ignoring epochs [127,127] <= 127 2014-08-22 19:26:45,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.274536 7f25dc949700 20 client.4209.objecter dump_active .. 0 homeless 2014-08-22 19:26:45,945.945 INFO:tasks.rados.rados.0.mira097.stdout:892: done (0 left) 2014-08-22 19:26:45,945.945 INFO:tasks.rados.rados.0.mira097.stdout:893: delete oid 297 current snap is 42 2014-08-22 19:26:45,945.945 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 0.608508 (8398792) 2014-08-22 19:26:45,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.858314 7f25df89c700 20 client.4209.objecter shutdown clearing up homeless session... 2014-08-22 19:26:45,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:38.858918 7f25df89c700 1 librados: shutdown 2014-08-22 19:26:46,119.119 INFO:tasks.rados.rados.0.mira097.stdout:893: done (0 left) 2014-08-22 19:26:46,119.119 INFO:tasks.rados.rados.0.mira097.stdout:894: write oid 356 current snap is 42 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:894: seq_num 641 ranges {601561=682455,1401560=1} 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:894: writing mira09725123-356 from 601561 to 1284016 tid 1 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:894: writing mira09725123-356 from 1401560 to 1401561 tid 2 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:895: rollback oid 360 current snap is 42 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 360 to 32 2014-08-22 19:26:46,120.120 INFO:tasks.rados.rados.0.mira097.stdout:896: read oid 62 snap 38 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:897: rollback oid 381 current snap is 42 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 381 to 33 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:898: write oid 225 current snap is 42 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:898: seq_num 642 ranges {546391=405683,1551102=595289} 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:898: writing mira09725123-225 from 546391 to 952074 tid 1 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:898: writing mira09725123-225 from 1551102 to 2146391 tid 2 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:899: delete oid 111 current snap is 42 2014-08-22 19:26:46,121.121 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 360 v 1341 (ObjNum 359 snap 0 seq_num 359) dirty exists 2014-08-22 19:26:46,122.122 INFO:tasks.rados.rados.0.mira097.stdout:896: expect (ObjNum 61 snap 0 seq_num 61) 2014-08-22 19:26:46,207.207 INFO:tasks.rados.rados.0.mira097.stdout:898: finishing write tid 1 to mira09725123-225 2014-08-22 19:26:46,207.207 INFO:tasks.rados.rados.0.mira097.stdout:898: finishing write tid 2 to mira09725123-225 2014-08-22 19:26:46,207.207 INFO:tasks.rados.rados.0.mira097.stdout:898: finishing write tid 4 to mira09725123-225 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:898: finishing write tid 5 to mira09725123-225 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 225 v 1360 (ObjNum 642 snap 42 seq_num 642) dirty exists 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:895: done (5 left) 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:896: done (4 left) 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:898: done (3 left) 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:899: done (2 left) 2014-08-22 19:26:46,208.208 INFO:tasks.rados.rados.0.mira097.stdout:900: copy_from oid 269 from oid 121 current snap is 42 2014-08-22 19:26:46,209.209 INFO:tasks.rados.rados.0.mira097.stdout:901: copy_from oid 419 from oid 8 current snap is 42 2014-08-22 19:26:46,209.209 INFO:tasks.rados.rados.0.mira097.stdout:902: write oid 432 current snap is 42 2014-08-22 19:26:46,209.209 INFO:tasks.rados.rados.0.mira097.stdout:902: seq_num 644 ranges {586450=633964,1876518=515698,2586449=1} 2014-08-22 19:26:46,235.235 INFO:tasks.rados.rados.0.mira097.stdout:902: writing mira09725123-432 from 586450 to 1220414 tid 1 2014-08-22 19:26:46,256.256 INFO:tasks.rados.rados.0.mira097.stdout:902: writing mira09725123-432 from 1876518 to 2392216 tid 2 2014-08-22 19:26:46,261.261 INFO:tasks.rados.rados.0.mira097.stdout:902: writing mira09725123-432 from 2586449 to 2586450 tid 3 2014-08-22 19:26:46,262.262 INFO:tasks.rados.rados.0.mira097.stdout:903: read oid 211 snap 38 2014-08-22 19:26:46,262.262 INFO:tasks.rados.rados.0.mira097.stdout:902: finishing write tid 1 to mira09725123-432 2014-08-22 19:26:46,262.262 INFO:tasks.rados.rados.0.mira097.stdout:904: snap_remove snap 41 2014-08-22 19:26:46,273.273 INFO:tasks.rados.rados.0.mira097.stdout:902: finishing write tid 2 to mira09725123-432 2014-08-22 19:26:46,285.285 INFO:tasks.rados.rados.0.mira097.stdout:903: expect (ObjNum 503 snap 3 seq_num 503) 2014-08-22 19:26:47,238.238 INFO:tasks.rados.rados.0.mira097.stdout:902: finishing write tid 3 to mira09725123-432 2014-08-22 19:26:47,238.238 INFO:tasks.rados.rados.0.mira097.stdout:902: finishing write tid 5 to mira09725123-432 2014-08-22 19:26:47,238.238 INFO:tasks.rados.rados.0.mira097.stdout:903: done (6 left) 2014-08-22 19:26:47,238.238 INFO:tasks.rados.rados.0.mira097.stdout:904: done (5 left) 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:905: read oid 40 snap -1 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:902: finishing write tid 6 to mira09725123-432 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 432 v 1202 (ObjNum 644 snap 42 seq_num 644) dirty exists 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:902: done (5 left) 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:906: read oid 446 snap -1 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:907: rollback oid 165 current snap is 42 2014-08-22 19:26:47,239.239 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 165 to 32 2014-08-22 19:26:47,240.240 INFO:tasks.rados.rados.0.mira097.stdout:908: read oid 260 snap -1 2014-08-22 19:26:47,240.240 INFO:tasks.rados.rados.0.mira097.stdout:909: rollback oid 153 current snap is 42 2014-08-22 19:26:47,240.240 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 153 to 40 2014-08-22 19:26:47,240.240 INFO:tasks.rados.rados.0.mira097.stdout:910: write oid 388 current snap is 42 2014-08-22 19:26:47,411.411 INFO:tasks.rados.rados.0.mira097.stdout:910: seq_num 645 ranges {647632=605768,1954554=737317,3477293=370339} 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:910: writing mira09725123-388 from 647632 to 1253400 tid 1 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:910: writing mira09725123-388 from 1954554 to 2691871 tid 2 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:910: writing mira09725123-388 from 3477293 to 3847632 tid 3 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:911: write oid 250 current snap is 42 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:911: seq_num 646 ranges {590413=1} 2014-08-22 19:26:47,412.412 INFO:tasks.rados.rados.0.mira097.stdout:911: writing mira09725123-250 from 590413 to 590414 tid 1 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:912: write oid 290 current snap is 42 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:912: seq_num 647 ranges {286667=1} 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:912: writing mira09725123-290 from 286667 to 286668 tid 1 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:913: write oid 291 current snap is 42 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:913: seq_num 648 ranges {630473=400000} 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:913: writing mira09725123-291 from 630473 to 1030473 tid 1 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:914: snap_remove snap 36 2014-08-22 19:26:47,413.413 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 165 v 1354 (ObjNum 534 snap 8 seq_num 534) dirty exists 2014-08-22 19:26:47,414.414 INFO:tasks.rados.rados.0.mira097.stdout:905: expect (ObjNum 39 snap 0 seq_num 39) 2014-08-22 19:26:48,649.649 INFO:tasks.rados.rados.0.mira097.stdout:906: expect (ObjNum 527 snap 8 seq_num 527) 2014-08-22 19:26:48,814.814 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 153 v 1357 (ObjNum 152 snap 0 seq_num 152) dirty exists 2014-08-22 19:26:48,814.814 INFO:tasks.rados.rados.0.mira097.stdout:910: finishing write tid 1 to mira09725123-388 2014-08-22 19:26:48,814.814 INFO:tasks.rados.rados.0.mira097.stdout:910: finishing write tid 2 to mira09725123-388 2014-08-22 19:26:48,814.814 INFO:tasks.rados.rados.0.mira097.stdout:908: expect (ObjNum 259 snap 0 seq_num 259) 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:910: finishing write tid 3 to mira09725123-388 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:910: finishing write tid 5 to mira09725123-388 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:910: finishing write tid 6 to mira09725123-388 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 388 v 1369 (ObjNum 645 snap 42 seq_num 645) dirty exists 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:905: done (13 left) 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:906: done (12 left) 2014-08-22 19:26:49,705.705 INFO:tasks.rados.rados.0.mira097.stdout:907: done (11 left) 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:908: done (10 left) 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:909: done (9 left) 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:910: done (8 left) 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:914: done (7 left) 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:915: copy_from oid 29 from oid 62 current snap is 42 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:911: finishing write tid 1 to mira09725123-250 2014-08-22 19:26:49,706.706 INFO:tasks.rados.rados.0.mira097.stdout:911: finishing write tid 3 to mira09725123-250 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:912: finishing write tid 1 to mira09725123-290 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:912: finishing write tid 3 to mira09725123-290 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:911: finishing write tid 4 to mira09725123-250 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 250 v 1373 (ObjNum 646 snap 42 seq_num 646) dirty exists 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:912: finishing write tid 4 to mira09725123-290 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 290 v 1214 (ObjNum 647 snap 42 seq_num 647) dirty exists 2014-08-22 19:26:49,707.707 INFO:tasks.rados.rados.0.mira097.stdout:913: finishing write tid 1 to mira09725123-291 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:913: finishing write tid 3 to mira09725123-291 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:913: finishing write tid 4 to mira09725123-291 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 291 v 1362 (ObjNum 648 snap 42 seq_num 648) dirty exists 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:901: finishing copy_from to mira09725123-419 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 419 v 1043 (ObjNum 162 snap 0 seq_num 162) dirty exists 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:901: finishing copy_from racing read to mira09725123-419 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:894: finishing write tid 1 to mira09725123-356 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:894: finishing write tid 2 to mira09725123-356 2014-08-22 19:26:49,708.708 INFO:tasks.rados.rados.0.mira097.stdout:894: finishing write tid 4 to mira09725123-356 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:894: finishing write tid 5 to mira09725123-356 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 356 v 1319 (ObjNum 641 snap 42 seq_num 641) dirty exists 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 381 v 1323 (ObjNum 380 snap 0 seq_num 380) dirty exists 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:900: finishing copy_from to mira09725123-269 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 269 v 143 (ObjNum 120 snap 0 seq_num 120) dirty exists 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:900: finishing copy_from racing read to mira09725123-269 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:894: done (7 left) 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:897: done (6 left) 2014-08-22 19:26:49,709.709 INFO:tasks.rados.rados.0.mira097.stdout:900: done (5 left) 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout:901: done (4 left) 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout:911: done (3 left) 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout:912: done (2 left) 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout:913: done (1 left) 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout:916: snap_create 2014-08-22 19:26:49,710.710 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:49,745.745 INFO:tasks.rados.rados.0.mira097.stdout:915: finishing copy_from to mira09725123-29 2014-08-22 19:26:49,745.745 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 29 v 747 (ObjNum 61 snap 0 seq_num 61) dirty exists 2014-08-22 19:26:49,748.748 INFO:tasks.rados.rados.0.mira097.stdout:915: finishing copy_from racing read to mira09725123-29 2014-08-22 19:26:49,748.748 INFO:tasks.rados.rados.0.mira097.stdout:915: done (0 left) 2014-08-22 19:26:49,987.987 INFO:tasks.rados.rados.0.mira097.stdout:916: done (0 left) 2014-08-22 19:26:49,988.988 INFO:tasks.rados.rados.0.mira097.stdout:917: write oid 457 current snap is 43 2014-08-22 19:26:49,988.988 INFO:tasks.rados.rados.0.mira097.stdout:917: seq_num 649 ranges {575303=681530,1775302=1} 2014-08-22 19:26:50,019.019 INFO:tasks.rados.rados.0.mira097.stdout:917: writing mira09725123-457 from 575303 to 1256833 tid 1 2014-08-22 19:26:50,024.024 INFO:tasks.rados.rados.0.mira097.stdout:917: writing mira09725123-457 from 1775302 to 1775303 tid 2 2014-08-22 19:26:50,025.025 INFO:tasks.rados.rados.0.mira097.stdout:918: rollback oid 218 current snap is 43 2014-08-22 19:26:50,025.025 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 218 to 39 2014-08-22 19:26:50,025.025 INFO:tasks.rados.rados.0.mira097.stdout:919: write oid 361 current snap is 43 2014-08-22 19:26:50,025.025 INFO:tasks.rados.rados.0.mira097.stdout:919: seq_num 650 ranges {787909=405782,1655071=332838} 2014-08-22 19:26:50,040.040 INFO:tasks.rados.rados.0.mira097.stdout:919: writing mira09725123-361 from 787909 to 1193691 tid 1 2014-08-22 19:26:50,053.053 INFO:tasks.rados.rados.0.mira097.stdout:919: writing mira09725123-361 from 1655071 to 1987909 tid 2 2014-08-22 19:26:50,055.055 INFO:tasks.rados.rados.0.mira097.stdout:920: write oid 9 current snap is 43 2014-08-22 19:26:50,055.055 INFO:tasks.rados.rados.0.mira097.stdout:920: seq_num 651 ranges {731715=777586,2016755=543960,2731714=1} 2014-08-22 19:26:50,081.081 INFO:tasks.rados.rados.0.mira097.stdout:920: writing mira09725123-9 from 731715 to 1509301 tid 1 2014-08-22 19:26:50,098.098 INFO:tasks.rados.rados.0.mira097.stdout:920: writing mira09725123-9 from 2016755 to 2560715 tid 2 2014-08-22 19:26:50,102.102 INFO:tasks.rados.rados.0.mira097.stdout:920: writing mira09725123-9 from 2731714 to 2731715 tid 3 2014-08-22 19:26:50,104.104 INFO:tasks.rados.rados.0.mira097.stdout:921: write oid 142 current snap is 43 2014-08-22 19:26:50,104.104 INFO:tasks.rados.rados.0.mira097.stdout:921: seq_num 652 ranges {427968=784463,1882312=545656} 2014-08-22 19:26:50,125.125 INFO:tasks.rados.rados.0.mira097.stdout:921: writing mira09725123-142 from 427968 to 1212431 tid 1 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:921: writing mira09725123-142 from 1882312 to 2427968 tid 2 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:922: read oid 392 snap -1 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:917: finishing write tid 1 to mira09725123-457 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:917: finishing write tid 2 to mira09725123-457 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:917: finishing write tid 4 to mira09725123-457 2014-08-22 19:26:50,161.161 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 218 v 1382 (ObjNum 217 snap 0 seq_num 217) dirty exists 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:917: finishing write tid 5 to mira09725123-457 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 457 v 1380 (ObjNum 649 snap 43 seq_num 649) dirty exists 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:919: finishing write tid 1 to mira09725123-361 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:919: finishing write tid 2 to mira09725123-361 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:919: finishing write tid 4 to mira09725123-361 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:919: finishing write tid 5 to mira09725123-361 2014-08-22 19:26:50,162.162 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 361 v 1336 (ObjNum 650 snap 43 seq_num 650) dirty exists 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:920: finishing write tid 1 to mira09725123-9 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:920: finishing write tid 2 to mira09725123-9 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:920: finishing write tid 3 to mira09725123-9 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:920: finishing write tid 5 to mira09725123-9 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:917: done (5 left) 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:918: done (4 left) 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:919: done (3 left) 2014-08-22 19:26:50,163.163 INFO:tasks.rados.rados.0.mira097.stdout:923: delete oid 243 current snap is 43 2014-08-22 19:26:50,192.192 INFO:tasks.rados.rados.0.mira097.stdout:923: done (3 left) 2014-08-22 19:26:50,192.192 INFO:tasks.rados.rados.0.mira097.stdout:924: rollback oid 257 current snap is 43 2014-08-22 19:26:50,192.192 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 257 to 32 2014-08-22 19:26:50,192.192 INFO:tasks.rados.rados.0.mira097.stdout:925: read oid 355 snap -1 2014-08-22 19:26:50,193.193 INFO:tasks.rados.rados.0.mira097.stdout:926: read oid 204 snap -1 2014-08-22 19:26:50,193.193 INFO:tasks.rados.rados.0.mira097.stdout:927: read oid 52 snap 38 2014-08-22 19:26:50,193.193 INFO:tasks.rados.rados.0.mira097.stdout:928: delete oid 51 current snap is 43 2014-08-22 19:26:50,260.260 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 257 v 1223 (ObjNum 37 snap 0 seq_num 37) dirty exists 2014-08-22 19:26:50,290.290 INFO:tasks.rados.rados.0.mira097.stdout:927: expect (ObjNum 562 snap 20 seq_num 562) 2014-08-22 19:26:50,977.977 INFO:tasks.rados.rados.0.mira097.stdout:920: finishing write tid 6 to mira09725123-9 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 9 v 1388 (ObjNum 651 snap 43 seq_num 651) dirty exists 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:921: finishing write tid 1 to mira09725123-142 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:921: finishing write tid 2 to mira09725123-142 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:921: finishing write tid 4 to mira09725123-142 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:920: done (7 left) 2014-08-22 19:26:50,978.978 INFO:tasks.rados.rados.0.mira097.stdout:924: done (6 left) 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:927: done (5 left) 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:928: done (4 left) 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:929: copy_from oid 354 from oid 268 current snap is 43 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:921: finishing write tid 5 to mira09725123-142 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 142 v 1392 (ObjNum 652 snap 43 seq_num 652) dirty exists 2014-08-22 19:26:50,979.979 INFO:tasks.rados.rados.0.mira097.stdout:925: expect (ObjNum 354 snap 0 seq_num 354) 2014-08-22 19:26:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:922: expect (ObjNum 391 snap 0 seq_num 391) 2014-08-22 19:26:51,118.118 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:51,118.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:51,119.119 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.863979016126 2014-08-22 19:26:51,119.119 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 0.863979016126' 2014-08-22 19:26:51,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.488982 7fd724e90700 1 librados: starting msgr at :/0 2014-08-22 19:26:51,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.488987 7fd724e90700 1 librados: starting objecter 2014-08-22 19:26:51,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.489321 7fd724e90700 1 librados: setting wanted keys 2014-08-22 19:26:51,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.489326 7fd724e90700 1 librados: calling monclient init 2014-08-22 19:26:51,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.490985 7fd71dd8a700 10 client.?.objecter ms_handle_connect 0x7fd7200355e0 2014-08-22 19:26:51,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.491008 7fd71dd8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:51,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.493946 7fd724e90700 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:51,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.494046 7fd724e90700 1 librados: init done 2014-08-22 19:26:51,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.494956 7fd71dd8a700 10 client.4213.objecter ms_dispatch 0x7fd720000f80 osd_map(131..131 src has 1..131) v3 2014-08-22 19:26:51,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.494966 7fd71dd8a700 3 client.4213.objecter handle_osd_map got epochs [131,131] > 0 2014-08-22 19:26:51,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.494969 7fd71dd8a700 3 client.4213.objecter handle_osd_map decoding full epoch 131 2014-08-22 19:26:51,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.495057 7fd71dd8a700 20 client.4213.objecter dump_active .. 0 homeless 2014-08-22 19:26:51,686.686 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.495197 7fd71dd8a700 10 client.4213.objecter ms_dispatch 0x7fd720000f80 osd_map(131..131 src has 1..131) v3 2014-08-22 19:26:51,686.686 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.495202 7fd71dd8a700 3 client.4213.objecter handle_osd_map ignoring epochs [131,131] <= 131 2014-08-22 19:26:51,686.686 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.495204 7fd71dd8a700 20 client.4213.objecter dump_active .. 0 homeless 2014-08-22 19:26:51,833.833 INFO:tasks.rados.rados.0.mira097.stdout:926: expect (ObjNum 203 snap 0 seq_num 203) 2014-08-22 19:26:52,030.030 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 0.863979 (8566212) 2014-08-22 19:26:52,034.034 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.940532 7fd724e90700 20 client.4213.objecter shutdown clearing up homeless session... 2014-08-22 19:26:52,034.034 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:44.940922 7fd724e90700 1 librados: shutdown 2014-08-22 19:26:53,557.557 INFO:tasks.rados.rados.0.mira097.stdout:929: finishing copy_from to mira09725123-354 2014-08-22 19:26:53,557.557 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 354 v 1021 (ObjNum 267 snap 0 seq_num 267) dirty exists 2014-08-22 19:26:53,557.557 INFO:tasks.rados.rados.0.mira097.stdout:929: finishing copy_from racing read to mira09725123-354 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:921: done (4 left) 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:922: done (3 left) 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:925: done (2 left) 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:926: done (1 left) 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:929: done (0 left) 2014-08-22 19:26:53,558.558 INFO:tasks.rados.rados.0.mira097.stdout:930: snap_create 2014-08-22 19:26:54,111.111 INFO:tasks.rados.rados.0.mira097.stdout:930: done (0 left) 2014-08-22 19:26:54,111.111 INFO:tasks.rados.rados.0.mira097.stdout:931: write oid 445 current snap is 44 2014-08-22 19:26:54,111.111 INFO:tasks.rados.rados.0.mira097.stdout:931: seq_num 655 ranges {128184=1} 2014-08-22 19:26:54,112.112 INFO:tasks.rados.rados.0.mira097.stdout:931: writing mira09725123-445 from 128184 to 128185 tid 1 2014-08-22 19:26:54,113.113 INFO:tasks.rados.rados.0.mira097.stdout:932: write oid 94 current snap is 44 2014-08-22 19:26:54,113.113 INFO:tasks.rados.rados.0.mira097.stdout:932: seq_num 656 ranges {471991=400000} 2014-08-22 19:26:54,136.136 INFO:tasks.rados.rados.0.mira097.stdout:932: writing mira09725123-94 from 471991 to 871991 tid 1 2014-08-22 19:26:54,137.137 INFO:tasks.rados.rados.0.mira097.stdout:933: snap_create 2014-08-22 19:26:54,137.137 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:54,138.138 INFO:tasks.rados.rados.0.mira097.stdout:931: finishing write tid 1 to mira09725123-445 2014-08-22 19:26:54,138.138 INFO:tasks.rados.rados.0.mira097.stdout:931: finishing write tid 3 to mira09725123-445 2014-08-22 19:26:54,138.138 INFO:tasks.rados.rados.0.mira097.stdout:931: finishing write tid 4 to mira09725123-445 2014-08-22 19:26:54,139.139 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 445 v 1356 (ObjNum 655 snap 44 seq_num 655) dirty exists 2014-08-22 19:26:54,140.140 INFO:tasks.rados.rados.0.mira097.stdout:931: done (1 left) 2014-08-22 19:26:54,140.140 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:26:54,282.282 INFO:tasks.rados.rados.0.mira097.stdout:932: finishing write tid 1 to mira09725123-94 2014-08-22 19:26:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:932: finishing write tid 3 to mira09725123-94 2014-08-22 19:26:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:932: finishing write tid 4 to mira09725123-94 2014-08-22 19:26:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 94 v 1408 (ObjNum 656 snap 44 seq_num 656) dirty exists 2014-08-22 19:26:54,283.283 INFO:tasks.rados.rados.0.mira097.stdout:932: done (0 left) 2014-08-22 19:26:55,121.121 INFO:tasks.rados.rados.0.mira097.stdout:933: done (0 left) 2014-08-22 19:26:55,121.121 INFO:tasks.rados.rados.0.mira097.stdout:934: read oid 287 snap 33 2014-08-22 19:26:55,121.121 INFO:tasks.rados.rados.0.mira097.stdout:935: read oid 396 snap -1 2014-08-22 19:26:55,123.123 INFO:tasks.rados.rados.0.mira097.stdout:936: copy_from oid 145 from oid 82 current snap is 45 2014-08-22 19:26:55,123.123 INFO:tasks.rados.rados.0.mira097.stdout:934: done (2 left) 2014-08-22 19:26:55,123.123 INFO:tasks.rados.rados.0.mira097.stdout:937: write oid 139 current snap is 45 2014-08-22 19:26:55,123.123 INFO:tasks.rados.rados.0.mira097.stdout:937: seq_num 657 ranges {416821=679580,1525484=91337} 2014-08-22 19:26:55,151.151 INFO:tasks.rados.rados.0.mira097.stdout:937: writing mira09725123-139 from 416821 to 1096401 tid 1 2014-08-22 19:26:55,158.158 INFO:tasks.rados.rados.0.mira097.stdout:937: writing mira09725123-139 from 1525484 to 1616821 tid 2 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:938: read oid 118 snap -1 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:936: finishing copy_from to mira09725123-145 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:936: got expected ENOENT (src dne) 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:936: finishing copy_from racing read to mira09725123-145 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:936: done (3 left) 2014-08-22 19:26:55,159.159 INFO:tasks.rados.rados.0.mira097.stdout:939: read oid 234 snap -1 2014-08-22 19:26:55,160.160 INFO:tasks.rados.rados.0.mira097.stdout:940: snap_create 2014-08-22 19:26:55,160.160 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:26:55,174.174 INFO:tasks.rados.rados.0.mira097.stdout:939: expect (ObjNum 567 snap 20 seq_num 567) 2014-08-22 19:26:55,329.329 INFO:tasks.rados.rados.0.mira097.stdout:937: finishing write tid 1 to mira09725123-139 2014-08-22 19:26:55,330.330 INFO:tasks.rados.rados.0.mira097.stdout:937: finishing write tid 2 to mira09725123-139 2014-08-22 19:26:55,330.330 INFO:tasks.rados.rados.0.mira097.stdout:937: finishing write tid 4 to mira09725123-139 2014-08-22 19:26:55,330.330 INFO:tasks.rados.rados.0.mira097.stdout:937: finishing write tid 5 to mira09725123-139 2014-08-22 19:26:55,330.330 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 139 v 1402 (ObjNum 657 snap 45 seq_num 657) dirty exists 2014-08-22 19:26:55,330.330 INFO:tasks.rados.rados.0.mira097.stdout:938: expect (ObjNum 117 snap 0 seq_num 117) 2014-08-22 19:26:55,492.492 INFO:tasks.rados.rados.0.mira097.stdout:935: expect (ObjNum 395 snap 0 seq_num 395) 2014-08-22 19:26:55,661.661 INFO:tasks.rados.rados.0.mira097.stdout:935: done (3 left) 2014-08-22 19:26:55,661.661 INFO:tasks.rados.rados.0.mira097.stdout:937: done (2 left) 2014-08-22 19:26:55,661.661 INFO:tasks.rados.rados.0.mira097.stdout:938: done (1 left) 2014-08-22 19:26:55,661.661 INFO:tasks.rados.rados.0.mira097.stdout:939: done (0 left) 2014-08-22 19:26:56,138.138 INFO:tasks.rados.rados.0.mira097.stdout:940: done (0 left) 2014-08-22 19:26:56,138.138 INFO:tasks.rados.rados.0.mira097.stdout:941: read oid 449 snap -1 2014-08-22 19:26:56,139.139 INFO:tasks.rados.rados.0.mira097.stdout:942: write oid 106 current snap is 46 2014-08-22 19:26:56,139.139 INFO:tasks.rados.rados.0.mira097.stdout:942: seq_num 658 ranges {512051=403833,1312050=1} 2014-08-22 19:26:56,156.156 INFO:tasks.rados.rados.0.mira097.stdout:942: writing mira09725123-106 from 512051 to 915884 tid 1 2014-08-22 19:26:56,157.157 INFO:tasks.rados.rados.0.mira097.stdout:942: writing mira09725123-106 from 1312050 to 1312051 tid 2 2014-08-22 19:26:56,158.158 INFO:tasks.rados.rados.0.mira097.stdout:943: snap_create 2014-08-22 19:26:56,158.158 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:26:56,169.169 INFO:tasks.rados.rados.0.mira097.stdout:941: expect (ObjNum 448 snap 0 seq_num 448) 2014-08-22 19:26:57,039.039 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:26:57,039.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:26:57,040.040 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:26:57,040.040 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:26:57,277.277 INFO:tasks.rados.rados.0.mira097.stdout:942: finishing write tid 1 to mira09725123-106 2014-08-22 19:26:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:942: finishing write tid 2 to mira09725123-106 2014-08-22 19:26:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:942: finishing write tid 4 to mira09725123-106 2014-08-22 19:26:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:942: finishing write tid 5 to mira09725123-106 2014-08-22 19:26:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 106 v 1361 (ObjNum 658 snap 46 seq_num 658) dirty exists 2014-08-22 19:26:57,278.278 INFO:tasks.rados.rados.0.mira097.stdout:941: done (1 left) 2014-08-22 19:26:57,279.279 INFO:tasks.rados.rados.0.mira097.stdout:942: done (0 left) 2014-08-22 19:26:57,518.518 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.423942 7f1e457d2700 1 librados: starting msgr at :/0 2014-08-22 19:26:57,518.518 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.423950 7f1e457d2700 1 librados: starting objecter 2014-08-22 19:26:57,519.519 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.424191 7f1e457d2700 1 librados: setting wanted keys 2014-08-22 19:26:57,519.519 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.424201 7f1e457d2700 1 librados: calling monclient init 2014-08-22 19:26:57,520.520 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.426562 7f1e3e7fc700 10 client.?.objecter ms_handle_connect 0x7f1e400355e0 2014-08-22 19:26:57,520.520 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.426602 7f1e3e7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:57,524.524 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.430694 7f1e457d2700 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:57,524.524 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.430853 7f1e457d2700 1 librados: init done 2014-08-22 19:26:57,525.525 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431724 7f1e3e7fc700 10 client.4216.objecter ms_dispatch 0x7f1e40000f80 osd_map(136..136 src has 1..136) v3 2014-08-22 19:26:57,525.525 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431737 7f1e3e7fc700 3 client.4216.objecter handle_osd_map got epochs [136,136] > 0 2014-08-22 19:26:57,525.525 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431741 7f1e3e7fc700 3 client.4216.objecter handle_osd_map decoding full epoch 136 2014-08-22 19:26:57,525.525 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431883 7f1e3e7fc700 20 client.4216.objecter dump_active .. 0 homeless 2014-08-22 19:26:57,526.526 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431932 7f1e3e7fc700 10 client.4216.objecter ms_dispatch 0x7f1e40000f80 osd_map(136..136 src has 1..136) v3 2014-08-22 19:26:57,526.526 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431944 7f1e3e7fc700 3 client.4216.objecter handle_osd_map ignoring epochs [136,136] <= 136 2014-08-22 19:26:57,526.526 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.431947 7f1e3e7fc700 20 client.4216.objecter dump_active .. 0 homeless 2014-08-22 19:26:57,699.699 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:26:57,981.981 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.538810 7f1e457d2700 20 client.4216.objecter shutdown clearing up homeless session... 2014-08-22 19:26:57,981.981 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:50.539851 7f1e457d2700 1 librados: shutdown 2014-08-22 19:26:58,069.069 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 64' 2014-08-22 19:26:58,229.229 INFO:tasks.rados.rados.0.mira097.stdout:943: done (0 left) 2014-08-22 19:26:58,229.229 INFO:tasks.rados.rados.0.mira097.stdout:944: rollback oid 59 current snap is 47 2014-08-22 19:26:58,229.229 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 59 to 38 2014-08-22 19:26:58,230.230 INFO:tasks.rados.rados.0.mira097.stdout:945: read oid 326 snap -1 2014-08-22 19:26:58,230.230 INFO:tasks.rados.rados.0.mira097.stdout:946: copy_from oid 276 from oid 242 current snap is 47 2014-08-22 19:26:58,230.230 INFO:tasks.rados.rados.0.mira097.stdout:947: copy_from oid 166 from oid 261 current snap is 47 2014-08-22 19:26:58,230.230 INFO:tasks.rados.rados.0.mira097.stdout:948: delete oid 199 current snap is 47 2014-08-22 19:26:58,252.252 INFO:tasks.rados.rados.0.mira097.stdout:945: expect (ObjNum 325 snap 0 seq_num 325) 2014-08-22 19:26:58,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.595150 7f4037e60700 1 librados: starting msgr at :/0 2014-08-22 19:26:58,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.595155 7f4037e60700 1 librados: starting objecter 2014-08-22 19:26:58,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.595477 7f4037e60700 1 librados: setting wanted keys 2014-08-22 19:26:58,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.595482 7f4037e60700 1 librados: calling monclient init 2014-08-22 19:26:58,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.596431 7f4034f0d700 10 client.?.objecter ms_handle_connect 0x7f40300355e0 2014-08-22 19:26:58,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.596482 7f4034f0d700 10 client.?.objecter resend_mon_ops 2014-08-22 19:26:58,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598196 7f4037e60700 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:26:58,691.691 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598278 7f4037e60700 1 librados: init done 2014-08-22 19:26:58,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598771 7f4034f0d700 10 client.4212.objecter ms_dispatch 0x7f4030000f80 osd_map(137..137 src has 1..137) v3 2014-08-22 19:26:58,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598779 7f4034f0d700 3 client.4212.objecter handle_osd_map got epochs [137,137] > 0 2014-08-22 19:26:58,692.692 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598781 7f4034f0d700 3 client.4212.objecter handle_osd_map decoding full epoch 137 2014-08-22 19:26:58,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598855 7f4034f0d700 20 client.4212.objecter dump_active .. 0 homeless 2014-08-22 19:26:58,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598883 7f4034f0d700 10 client.4212.objecter ms_dispatch 0x7f4030000f80 osd_map(137..137 src has 1..137) v3 2014-08-22 19:26:58,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598886 7f4034f0d700 3 client.4212.objecter handle_osd_map ignoring epochs [137,137] <= 137 2014-08-22 19:26:58,693.693 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:51.598887 7f4034f0d700 20 client.4212.objecter dump_active .. 0 homeless 2014-08-22 19:27:00,026.026 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 59 v 1368 (ObjNum 58 snap 0 seq_num 58) dirty exists 2014-08-22 19:27:00,026.026 INFO:tasks.rados.rados.0.mira097.stdout:947: finishing copy_from to mira09725123-166 2014-08-22 19:27:00,026.026 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 166 v 337 (ObjNum 260 snap 0 seq_num 260) dirty exists 2014-08-22 19:27:00,026.026 INFO:tasks.rados.rados.0.mira097.stdout:947: finishing copy_from racing read to mira09725123-166 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:946: finishing copy_from to mira09725123-276 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 276 v 298 (ObjNum 241 snap 0 seq_num 241) dirty exists 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:946: finishing copy_from racing read to mira09725123-276 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:944: done (4 left) 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:945: done (3 left) 2014-08-22 19:27:00,027.027 INFO:tasks.rados.rados.0.mira097.stdout:946: done (2 left) 2014-08-22 19:27:00,041.041 INFO:tasks.rados.rados.0.mira097.stdout:947: done (1 left) 2014-08-22 19:27:00,041.041 INFO:tasks.rados.rados.0.mira097.stdout:948: done (0 left) 2014-08-22 19:27:00,041.041 INFO:tasks.rados.rados.0.mira097.stdout:949: read oid 185 snap -1 2014-08-22 19:27:00,042.042 INFO:tasks.rados.rados.0.mira097.stdout:950: snap_create 2014-08-22 19:27:00,042.042 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:00,042.042 INFO:tasks.rados.rados.0.mira097.stdout:949: expect (ObjNum 184 snap 0 seq_num 184) 2014-08-22 19:27:00,042.042 INFO:tasks.rados.rados.0.mira097.stdout:949: done (0 left) 2014-08-22 19:27:00,043.043 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 64 2014-08-22 19:27:00,043.043 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:52.152188 7f4037e60700 20 client.4212.objecter shutdown clearing up homeless session... 2014-08-22 19:27:00,043.043 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:52.152524 7f4037e60700 1 librados: shutdown 2014-08-22 19:27:00,335.335 INFO:tasks.rados.rados.0.mira097.stdout:950: done (0 left) 2014-08-22 19:27:00,335.335 INFO:tasks.rados.rados.0.mira097.stdout:951: write oid 477 current snap is 48 2014-08-22 19:27:00,335.335 INFO:tasks.rados.rados.0.mira097.stdout:951: seq_num 660 ranges {518062=782513,1832808=712613,3104825=213237} 2014-08-22 19:27:00,384.384 INFO:tasks.rados.rados.0.mira097.stdout:951: writing mira09725123-477 from 518062 to 1300575 tid 1 2014-08-22 19:27:00,413.413 INFO:tasks.rados.rados.0.mira097.stdout:951: writing mira09725123-477 from 1832808 to 2545421 tid 2 2014-08-22 19:27:00,424.424 INFO:tasks.rados.rados.0.mira097.stdout:951: writing mira09725123-477 from 3104825 to 3318062 tid 3 2014-08-22 19:27:00,424.424 INFO:tasks.rados.rados.0.mira097.stdout:952: delete oid 416 current snap is 48 2014-08-22 19:27:00,425.425 INFO:tasks.rados.rados.0.mira097.stdout:951: finishing write tid 1 to mira09725123-477 2014-08-22 19:27:00,428.428 INFO:tasks.rados.rados.0.mira097.stdout:951: finishing write tid 2 to mira09725123-477 2014-08-22 19:27:00,434.434 INFO:tasks.rados.rados.0.mira097.stdout:952: done (1 left) 2014-08-22 19:27:00,438.438 INFO:tasks.rados.rados.0.mira097.stdout:953: write oid 315 current snap is 48 2014-08-22 19:27:00,439.439 INFO:tasks.rados.rados.0.mira097.stdout:953: seq_num 662 ranges {708522=478570,1776401=604427,2708521=1} 2014-08-22 19:27:00,458.458 INFO:tasks.rados.rados.0.mira097.stdout:953: writing mira09725123-315 from 708522 to 1187092 tid 1 2014-08-22 19:27:00,480.480 INFO:tasks.rados.rados.0.mira097.stdout:953: writing mira09725123-315 from 1776401 to 2380828 tid 2 2014-08-22 19:27:00,483.483 INFO:tasks.rados.rados.0.mira097.stdout:953: writing mira09725123-315 from 2708521 to 2708522 tid 3 2014-08-22 19:27:00,484.484 INFO:tasks.rados.rados.0.mira097.stdout:954: delete oid 388 current snap is 48 2014-08-22 19:27:00,484.484 INFO:tasks.rados.rados.0.mira097.stdout:951: finishing write tid 3 to mira09725123-477 2014-08-22 19:27:00,484.484 INFO:tasks.rados.rados.0.mira097.stdout:951: finishing write tid 5 to mira09725123-477 2014-08-22 19:27:00,484.484 INFO:tasks.rados.rados.0.mira097.stdout:951: finishing write tid 6 to mira09725123-477 2014-08-22 19:27:00,484.484 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 477 v 1236 (ObjNum 660 snap 48 seq_num 660) dirty exists 2014-08-22 19:27:00,491.491 INFO:tasks.rados.rados.0.mira097.stdout:953: finishing write tid 1 to mira09725123-315 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:953: finishing write tid 2 to mira09725123-315 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:953: finishing write tid 3 to mira09725123-315 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:953: finishing write tid 5 to mira09725123-315 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:951: done (2 left) 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:954: done (1 left) 2014-08-22 19:27:00,505.505 INFO:tasks.rados.rados.0.mira097.stdout:955: read oid 275 snap 32 2014-08-22 19:27:00,506.506 INFO:tasks.rados.rados.0.mira097.stdout:956: delete oid 77 current snap is 48 2014-08-22 19:27:00,506.506 INFO:tasks.rados.rados.0.mira097.stdout:953: finishing write tid 6 to mira09725123-315 2014-08-22 19:27:00,507.507 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 315 v 1425 (ObjNum 662 snap 48 seq_num 662) dirty exists 2014-08-22 19:27:00,640.640 INFO:tasks.rados.rados.0.mira097.stdout:953: done (2 left) 2014-08-22 19:27:00,640.640 INFO:tasks.rados.rados.0.mira097.stdout:956: done (1 left) 2014-08-22 19:27:00,640.640 INFO:tasks.rados.rados.0.mira097.stdout:957: read oid 84 snap -1 2014-08-22 19:27:00,641.641 INFO:tasks.rados.rados.0.mira097.stdout:955: expect (ObjNum 274 snap 0 seq_num 274) 2014-08-22 19:27:01,262.262 INFO:tasks.rados.rados.0.mira097.stdout:957: expect (ObjNum 619 snap 31 seq_num 619) 2014-08-22 19:27:02,219.219 INFO:tasks.rados.rados.0.mira097.stdout:955: done (1 left) 2014-08-22 19:27:02,220.220 INFO:tasks.rados.rados.0.mira097.stdout:957: done (0 left) 2014-08-22 19:27:02,220.220 INFO:tasks.rados.rados.0.mira097.stdout:958: copy_from oid 244 from oid 367 current snap is 48 2014-08-22 19:27:02,220.220 INFO:tasks.rados.rados.0.mira097.stdout:959: delete oid 149 current snap is 48 2014-08-22 19:27:02,243.243 INFO:tasks.rados.rados.0.mira097.stdout:959: done (1 left) 2014-08-22 19:27:02,243.243 INFO:tasks.rados.rados.0.mira097.stdout:960: copy_from oid 399 from oid 94 current snap is 48 2014-08-22 19:27:02,243.243 INFO:tasks.rados.rados.0.mira097.stdout:961: snap_create 2014-08-22 19:27:02,243.243 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:27:02,260.260 INFO:tasks.rados.rados.0.mira097.stdout:960: finishing copy_from to mira09725123-399 2014-08-22 19:27:02,260.260 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 399 v 1408 (ObjNum 656 snap 44 seq_num 656) dirty exists 2014-08-22 19:27:02,555.555 INFO:tasks.rados.rados.0.mira097.stdout:960: finishing copy_from racing read to mira09725123-399 2014-08-22 19:27:02,555.555 INFO:tasks.rados.rados.0.mira097.stdout:960: done (1 left) 2014-08-22 19:27:02,555.555 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:02,624.624 INFO:tasks.rados.rados.0.mira097.stdout:958: finishing copy_from to mira09725123-244 2014-08-22 19:27:02,624.624 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 244 v 500 (ObjNum 366 snap 0 seq_num 366) dirty exists 2014-08-22 19:27:02,624.624 INFO:tasks.rados.rados.0.mira097.stdout:958: finishing copy_from racing read to mira09725123-244 2014-08-22 19:27:02,624.624 INFO:tasks.rados.rados.0.mira097.stdout:958: done (0 left) 2014-08-22 19:27:02,685.685 INFO:tasks.rados.rados.0.mira097.stdout:961: done (0 left) 2014-08-22 19:27:02,685.685 INFO:tasks.rados.rados.0.mira097.stdout:962: snap_remove snap 40 2014-08-22 19:27:03,693.693 INFO:tasks.rados.rados.0.mira097.stdout:962: done (0 left) 2014-08-22 19:27:03,694.694 INFO:tasks.rados.rados.0.mira097.stdout:963: read oid 263 snap -1 2014-08-22 19:27:03,694.694 INFO:tasks.rados.rados.0.mira097.stdout:964: snap_create 2014-08-22 19:27:03,694.694 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:03,739.739 INFO:tasks.rados.rados.0.mira097.stdout:963: expect (ObjNum 583 snap 25 seq_num 583) 2014-08-22 19:27:03,780.780 INFO:tasks.rados.rados.0.mira097.stdout:963: done (0 left) 2014-08-22 19:27:04,703.703 INFO:tasks.rados.rados.0.mira097.stdout:964: done (0 left) 2014-08-22 19:27:04,703.703 INFO:tasks.rados.rados.0.mira097.stdout:965: write oid 134 current snap is 50 2014-08-22 19:27:04,703.703 INFO:tasks.rados.rados.0.mira097.stdout:965: seq_num 666 ranges {602145=401883,1742561=459584} 2014-08-22 19:27:04,725.725 INFO:tasks.rados.rados.0.mira097.stdout:965: writing mira09725123-134 from 602145 to 1004028 tid 1 2014-08-22 19:27:04,745.745 INFO:tasks.rados.rados.0.mira097.stdout:965: writing mira09725123-134 from 1742561 to 2202145 tid 2 2014-08-22 19:27:04,746.746 INFO:tasks.rados.rados.0.mira097.stdout:966: read oid 460 snap -1 2014-08-22 19:27:04,749.749 INFO:tasks.rados.rados.0.mira097.stdout:965: finishing write tid 1 to mira09725123-134 2014-08-22 19:27:04,749.749 INFO:tasks.rados.rados.0.mira097.stdout:967: write oid 290 current snap is 50 2014-08-22 19:27:04,749.749 INFO:tasks.rados.rados.0.mira097.stdout:967: seq_num 667 ranges {698398=773688,1898397=1} 2014-08-22 19:27:04,775.775 INFO:tasks.rados.rados.0.mira097.stdout:967: writing mira09725123-290 from 698398 to 1472086 tid 1 2014-08-22 19:27:04,778.778 INFO:tasks.rados.rados.0.mira097.stdout:967: writing mira09725123-290 from 1898397 to 1898398 tid 2 2014-08-22 19:27:04,778.778 INFO:tasks.rados.rados.0.mira097.stdout:968: delete oid 187 current snap is 50 2014-08-22 19:27:04,778.778 INFO:tasks.rados.rados.0.mira097.stdout:965: finishing write tid 2 to mira09725123-134 2014-08-22 19:27:04,779.779 INFO:tasks.rados.rados.0.mira097.stdout:965: finishing write tid 4 to mira09725123-134 2014-08-22 19:27:04,779.779 INFO:tasks.rados.rados.0.mira097.stdout:965: finishing write tid 5 to mira09725123-134 2014-08-22 19:27:04,876.876 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 134 v 1392 (ObjNum 666 snap 50 seq_num 666) dirty exists 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:965: done (3 left) 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:968: done (2 left) 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:969: write oid 33 current snap is 50 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:969: seq_num 669 ranges {703386=753393,1862367=482289,2807311=429255,3865129=38257} 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:969: writing mira09725123-33 from 703386 to 1456779 tid 1 2014-08-22 19:27:04,877.877 INFO:tasks.rados.rados.0.mira097.stdout:969: writing mira09725123-33 from 1862367 to 2344656 tid 2 2014-08-22 19:27:04,878.878 INFO:tasks.rados.rados.0.mira097.stdout:969: writing mira09725123-33 from 2807311 to 3236566 tid 3 2014-08-22 19:27:04,878.878 INFO:tasks.rados.rados.0.mira097.stdout:969: writing mira09725123-33 from 3865129 to 3903386 tid 4 2014-08-22 19:27:04,878.878 INFO:tasks.rados.rados.0.mira097.stdout:970: write oid 132 current snap is 50 2014-08-22 19:27:04,878.878 INFO:tasks.rados.rados.0.mira097.stdout:970: seq_num 670 ranges {798616=476621,1726899=703908,3159960=438656} 2014-08-22 19:27:04,882.882 INFO:tasks.rados.rados.0.mira097.stdout:970: writing mira09725123-132 from 798616 to 1275237 tid 1 2014-08-22 19:27:04,907.907 INFO:tasks.rados.rados.0.mira097.stdout:970: writing mira09725123-132 from 1726899 to 2430807 tid 2 2014-08-22 19:27:04,923.923 INFO:tasks.rados.rados.0.mira097.stdout:970: writing mira09725123-132 from 3159960 to 3598616 tid 3 2014-08-22 19:27:04,926.926 INFO:tasks.rados.rados.0.mira097.stdout:971: write oid 498 current snap is 50 2014-08-22 19:27:04,927.927 INFO:tasks.rados.rados.0.mira097.stdout:971: seq_num 671 ranges {493845=448425,1440006=774104,2995233=298612} 2014-08-22 19:27:04,939.939 INFO:tasks.rados.rados.0.mira097.stdout:971: writing mira09725123-498 from 493845 to 942270 tid 1 2014-08-22 19:27:04,965.965 INFO:tasks.rados.rados.0.mira097.stdout:971: writing mira09725123-498 from 1440006 to 2214110 tid 2 2014-08-22 19:27:04,984.984 INFO:tasks.rados.rados.0.mira097.stdout:971: writing mira09725123-498 from 2995233 to 3293845 tid 3 2014-08-22 19:27:04,988.988 INFO:tasks.rados.rados.0.mira097.stdout:972: copy_from oid 167 from oid 395 current snap is 50 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:967: finishing write tid 1 to mira09725123-290 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:967: finishing write tid 2 to mira09725123-290 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:967: finishing write tid 4 to mira09725123-290 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:967: finishing write tid 5 to mira09725123-290 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 290 v 1272 (ObjNum 667 snap 50 seq_num 667) dirty exists 2014-08-22 19:27:04,989.989 INFO:tasks.rados.rados.0.mira097.stdout:966: expect (ObjNum 459 snap 0 seq_num 459) 2014-08-22 19:27:05,026.026 INFO:tasks.thrashosds.thrasher:in_osds: [3, 5, 1, 2, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:05,027.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:05,027.027 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [3, 5, 1, 2, 0, 4] 2014-08-22 19:27:05,028.028 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-08-22 19:27:05,476.476 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.381646 7fb286e8d700 1 librados: starting msgr at :/0 2014-08-22 19:27:05,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.381652 7fb286e8d700 1 librados: starting objecter 2014-08-22 19:27:05,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.381794 7fb286e8d700 1 librados: setting wanted keys 2014-08-22 19:27:05,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.381801 7fb286e8d700 1 librados: calling monclient init 2014-08-22 19:27:05,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.382589 7fb2777fe700 10 client.?.objecter ms_handle_connect 0x7fb2800355e0 2014-08-22 19:27:05,477.477 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.382611 7fb2777fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:05,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.384440 7fb286e8d700 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:05,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.384533 7fb286e8d700 1 librados: init done 2014-08-22 19:27:05,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.384892 7fb2777fe700 10 client.4196.objecter ms_dispatch 0x7fb280000f80 osd_map(142..142 src has 1..142) v3 2014-08-22 19:27:05,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.384903 7fb2777fe700 3 client.4196.objecter handle_osd_map got epochs [142,142] > 0 2014-08-22 19:27:05,479.479 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.384906 7fb2777fe700 3 client.4196.objecter handle_osd_map decoding full epoch 142 2014-08-22 19:27:05,480.480 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.385014 7fb2777fe700 20 client.4196.objecter dump_active .. 0 homeless 2014-08-22 19:27:05,480.480 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.385063 7fb2777fe700 10 client.4196.objecter ms_dispatch 0x7fb280000f80 osd_map(142..142 src has 1..142) v3 2014-08-22 19:27:05,480.480 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.385067 7fb2777fe700 3 client.4196.objecter handle_osd_map ignoring epochs [142,142] <= 142 2014-08-22 19:27:05,480.480 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.385070 7fb2777fe700 20 client.4196.objecter dump_active .. 0 homeless 2014-08-22 19:27:05,768.768 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.5. 2014-08-22 19:27:05,771.771 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.677748 7fb286e8d700 20 client.4196.objecter shutdown clearing up homeless session... 2014-08-22 19:27:05,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:26:58.678274 7fb286e8d700 1 librados: shutdown 2014-08-22 19:27:06,690.690 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 1 to mira09725123-33 2014-08-22 19:27:06,690.690 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 2 to mira09725123-33 2014-08-22 19:27:06,690.690 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 3 to mira09725123-33 2014-08-22 19:27:06,690.690 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 4 to mira09725123-33 2014-08-22 19:27:06,690.690 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 6 to mira09725123-33 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:969: finishing write tid 7 to mira09725123-33 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 33 v 1426 (ObjNum 669 snap 50 seq_num 669) dirty exists 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:970: finishing write tid 1 to mira09725123-132 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:970: finishing write tid 2 to mira09725123-132 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:970: finishing write tid 3 to mira09725123-132 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:970: finishing write tid 5 to mira09725123-132 2014-08-22 19:27:06,691.691 INFO:tasks.rados.rados.0.mira097.stdout:970: finishing write tid 6 to mira09725123-132 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 132 v 1454 (ObjNum 670 snap 50 seq_num 670) dirty exists 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:971: finishing write tid 1 to mira09725123-498 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:971: finishing write tid 2 to mira09725123-498 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:971: finishing write tid 3 to mira09725123-498 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:971: finishing write tid 5 to mira09725123-498 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:971: finishing write tid 6 to mira09725123-498 2014-08-22 19:27:06,692.692 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 498 v 1283 (ObjNum 671 snap 50 seq_num 671) dirty exists 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:972: finishing copy_from to mira09725123-167 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 167 v 595 (ObjNum 394 snap 0 seq_num 394) dirty exists 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:972: finishing copy_from racing read to mira09725123-167 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:966: done (5 left) 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:967: done (4 left) 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:969: done (3 left) 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:970: done (2 left) 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:971: done (1 left) 2014-08-22 19:27:06,693.693 INFO:tasks.rados.rados.0.mira097.stdout:972: done (0 left) 2014-08-22 19:27:06,694.694 INFO:tasks.rados.rados.0.mira097.stdout:973: snap_create 2014-08-22 19:27:06,729.729 INFO:tasks.rados.rados.0.mira097.stdout:973: done (0 left) 2014-08-22 19:27:06,729.729 INFO:tasks.rados.rados.0.mira097.stdout:974: snap_remove snap 44 2014-08-22 19:27:07,759.759 INFO:tasks.rados.rados.0.mira097.stdout:974: done (0 left) 2014-08-22 19:27:07,759.759 INFO:tasks.rados.rados.0.mira097.stdout:975: delete oid 354 current snap is 51 2014-08-22 19:27:07,783.783 INFO:tasks.rados.rados.0.mira097.stdout:975: done (0 left) 2014-08-22 19:27:07,783.783 INFO:tasks.rados.rados.0.mira097.stdout:976: write oid 457 current snap is 51 2014-08-22 19:27:07,788.788 INFO:tasks.rados.rados.0.mira097.stdout:976: seq_num 673 ranges {499857=428130,1299856=1} 2014-08-22 19:27:07,807.807 INFO:tasks.rados.rados.0.mira097.stdout:976: writing mira09725123-457 from 499857 to 927987 tid 1 2014-08-22 19:27:07,813.813 INFO:tasks.rados.rados.0.mira097.stdout:976: writing mira09725123-457 from 1299856 to 1299857 tid 2 2014-08-22 19:27:07,819.819 INFO:tasks.rados.rados.0.mira097.stdout:977: write oid 196 current snap is 51 2014-08-22 19:27:07,819.819 INFO:tasks.rados.rados.0.mira097.stdout:977: seq_num 674 ranges {443662=551358,1712258=331404} 2014-08-22 19:27:07,833.833 INFO:tasks.rados.rados.0.mira097.stdout:977: writing mira09725123-196 from 443662 to 995020 tid 1 2014-08-22 19:27:07,844.844 INFO:tasks.rados.rados.0.mira097.stdout:977: writing mira09725123-196 from 1712258 to 2043662 tid 2 2014-08-22 19:27:07,850.850 INFO:tasks.rados.rados.0.mira097.stdout:978: write oid 375 current snap is 51 2014-08-22 19:27:07,850.850 INFO:tasks.rados.rados.0.mira097.stdout:978: seq_num 675 ranges {539916=772763,1739915=1} 2014-08-22 19:27:07,939.939 INFO:tasks.rados.rados.0.mira097.stdout:978: writing mira09725123-375 from 539916 to 1312679 tid 1 2014-08-22 19:27:07,939.939 INFO:tasks.rados.rados.0.mira097.stdout:978: writing mira09725123-375 from 1739915 to 1739916 tid 2 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:979: copy_from oid 56 from oid 217 current snap is 51 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:980: write oid 354 current snap is 51 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:980: seq_num 676 ranges {483722=780663,1922347=561375} 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:980: writing mira09725123-354 from 483722 to 1264385 tid 1 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:980: writing mira09725123-354 from 1922347 to 2483722 tid 2 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:981: read oid 336 snap -1 2014-08-22 19:27:07,940.940 INFO:tasks.rados.rados.0.mira097.stdout:978: finishing write tid 1 to mira09725123-375 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:978: finishing write tid 2 to mira09725123-375 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:978: finishing write tid 4 to mira09725123-375 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:978: finishing write tid 5 to mira09725123-375 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 375 v 1430 (ObjNum 675 snap 51 seq_num 675) dirty exists 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:977: finishing write tid 1 to mira09725123-196 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:977: finishing write tid 2 to mira09725123-196 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:977: finishing write tid 4 to mira09725123-196 2014-08-22 19:27:07,941.941 INFO:tasks.rados.rados.0.mira097.stdout:977: finishing write tid 5 to mira09725123-196 2014-08-22 19:27:07,942.942 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 196 v 1470 (ObjNum 674 snap 51 seq_num 674) dirty exists 2014-08-22 19:27:07,942.942 INFO:tasks.rados.rados.0.mira097.stdout:977: done (5 left) 2014-08-22 19:27:07,942.942 INFO:tasks.rados.rados.0.mira097.stdout:978: done (4 left) 2014-08-22 19:27:07,942.942 INFO:tasks.rados.rados.0.mira097.stdout:982: write oid 239 current snap is 51 2014-08-22 19:27:07,942.942 INFO:tasks.rados.rados.0.mira097.stdout:982: seq_num 677 ranges {544904=752467,1716735=580747,3046489=642939,3744903=1} 2014-08-22 19:27:07,956.956 INFO:tasks.rados.rados.0.mira097.stdout:982: writing mira09725123-239 from 544904 to 1297371 tid 1 2014-08-22 19:27:07,976.976 INFO:tasks.rados.rados.0.mira097.stdout:982: writing mira09725123-239 from 1716735 to 2297482 tid 2 2014-08-22 19:27:08,000.000 INFO:tasks.rados.rados.0.mira097.stdout:982: writing mira09725123-239 from 3046489 to 3689428 tid 3 2014-08-22 19:27:08,004.004 INFO:tasks.rados.rados.0.mira097.stdout:982: writing mira09725123-239 from 3744903 to 3744904 tid 4 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:983: copy_from oid 418 from oid 41 current snap is 51 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:976: finishing write tid 1 to mira09725123-457 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:976: finishing write tid 2 to mira09725123-457 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:976: finishing write tid 4 to mira09725123-457 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:976: finishing write tid 5 to mira09725123-457 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 457 v 1474 (ObjNum 673 snap 51 seq_num 673) dirty exists 2014-08-22 19:27:08,005.005 INFO:tasks.rados.rados.0.mira097.stdout:976: done (5 left) 2014-08-22 19:27:08,006.006 INFO:tasks.rados.rados.0.mira097.stdout:984: write oid 421 current snap is 51 2014-08-22 19:27:08,006.006 INFO:tasks.rados.rados.0.mira097.stdout:984: seq_num 678 ranges {640133=475696,1829843=650942,3247714=192419} 2014-08-22 19:27:08,019.019 INFO:tasks.rados.rados.0.mira097.stdout:984: writing mira09725123-421 from 640133 to 1115829 tid 1 2014-08-22 19:27:08,046.046 INFO:tasks.rados.rados.0.mira097.stdout:984: writing mira09725123-421 from 1829843 to 2480785 tid 2 2014-08-22 19:27:08,059.059 INFO:tasks.rados.rados.0.mira097.stdout:984: writing mira09725123-421 from 3247714 to 3440133 tid 3 2014-08-22 19:27:08,062.062 INFO:tasks.rados.rados.0.mira097.stdout:985: rollback oid 127 current snap is 51 2014-08-22 19:27:08,062.062 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 127 to 49 2014-08-22 19:27:08,063.063 INFO:tasks.rados.rados.0.mira097.stdout:986: write oid 315 current snap is 51 2014-08-22 19:27:08,063.063 INFO:tasks.rados.rados.0.mira097.stdout:986: seq_num 679 ranges {183938=1} 2014-08-22 19:27:08,063.063 INFO:tasks.rados.rados.0.mira097.stdout:986: writing mira09725123-315 from 183938 to 183939 tid 1 2014-08-22 19:27:08,063.063 INFO:tasks.rados.rados.0.mira097.stdout:987: snap_remove snap 38 2014-08-22 19:27:08,071.071 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 1 to mira09725123-239 2014-08-22 19:27:08,073.073 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 2 to mira09725123-239 2014-08-22 19:27:08,077.077 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 3 to mira09725123-239 2014-08-22 19:27:08,077.077 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 4 to mira09725123-239 2014-08-22 19:27:08,077.077 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 6 to mira09725123-239 2014-08-22 19:27:08,091.091 INFO:tasks.rados.rados.0.mira097.stdout:982: finishing write tid 7 to mira09725123-239 2014-08-22 19:27:08,091.091 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 239 v 1437 (ObjNum 677 snap 51 seq_num 677) dirty exists 2014-08-22 19:27:08,111.111 INFO:tasks.rados.rados.0.mira097.stdout:980: finishing write tid 1 to mira09725123-354 2014-08-22 19:27:08,115.115 INFO:tasks.rados.rados.0.mira097.stdout:980: finishing write tid 2 to mira09725123-354 2014-08-22 19:27:08,115.115 INFO:tasks.rados.rados.0.mira097.stdout:980: finishing write tid 4 to mira09725123-354 2014-08-22 19:27:08,118.118 INFO:tasks.rados.rados.0.mira097.stdout:980: finishing write tid 5 to mira09725123-354 2014-08-22 19:27:08,118.118 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 354 v 1410 (ObjNum 676 snap 51 seq_num 676) dirty exists 2014-08-22 19:27:08,149.149 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 127 v 1440 (ObjNum 126 snap 0 seq_num 126) dirty exists 2014-08-22 19:27:08,229.229 INFO:tasks.rados.rados.0.mira097.stdout:979: finishing copy_from to mira09725123-56 2014-08-22 19:27:08,229.229 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 56 v 299 (ObjNum 216 snap 0 seq_num 216) dirty exists 2014-08-22 19:27:08,242.242 INFO:tasks.rados.rados.0.mira097.stdout:984: finishing write tid 1 to mira09725123-421 2014-08-22 19:27:08,242.242 INFO:tasks.rados.rados.0.mira097.stdout:984: finishing write tid 2 to mira09725123-421 2014-08-22 19:27:08,242.242 INFO:tasks.rados.rados.0.mira097.stdout:984: finishing write tid 3 to mira09725123-421 2014-08-22 19:27:08,242.242 INFO:tasks.rados.rados.0.mira097.stdout:984: finishing write tid 5 to mira09725123-421 2014-08-22 19:27:08,246.246 INFO:tasks.rados.rados.0.mira097.stdout:986: finishing write tid 1 to mira09725123-315 2014-08-22 19:27:08,247.247 INFO:tasks.rados.rados.0.mira097.stdout:986: finishing write tid 3 to mira09725123-315 2014-08-22 19:27:08,261.261 INFO:tasks.rados.rados.0.mira097.stdout:979: finishing copy_from racing read to mira09725123-56 2014-08-22 19:27:08,300.300 INFO:tasks.rados.rados.0.mira097.stdout:984: finishing write tid 6 to mira09725123-421 2014-08-22 19:27:08,300.300 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 421 v 1427 (ObjNum 678 snap 51 seq_num 678) dirty exists 2014-08-22 19:27:08,340.340 INFO:tasks.rados.rados.0.mira097.stdout:983: finishing copy_from to mira09725123-418 2014-08-22 19:27:08,341.341 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 418 v 36 (ObjNum 40 snap 0 seq_num 40) dirty exists 2014-08-22 19:27:09,667.667 INFO:tasks.rados.rados.0.mira097.stdout:979: done (8 left) 2014-08-22 19:27:09,667.667 INFO:tasks.rados.rados.0.mira097.stdout:980: done (7 left) 2014-08-22 19:27:09,667.667 INFO:tasks.rados.rados.0.mira097.stdout:982: done (6 left) 2014-08-22 19:27:09,667.667 INFO:tasks.rados.rados.0.mira097.stdout:984: done (5 left) 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:985: done (4 left) 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:987: done (3 left) 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:988: read oid 21 snap -1 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:989: rollback oid 250 current snap is 51 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 250 to 46 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:990: delete oid 122 current snap is 51 2014-08-22 19:27:09,668.668 INFO:tasks.rados.rados.0.mira097.stdout:983: finishing copy_from racing read to mira09725123-418 2014-08-22 19:27:09,688.688 INFO:tasks.rados.rados.0.mira097.stdout:986: finishing write tid 4 to mira09725123-315 2014-08-22 19:27:09,688.688 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 315 v 1479 (ObjNum 679 snap 51 seq_num 679) dirty exists 2014-08-22 19:27:09,695.695 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 250 v 1442 (ObjNum 646 snap 42 seq_num 646) dirty exists 2014-08-22 19:27:09,740.740 INFO:tasks.rados.rados.0.mira097.stdout:988: expect (ObjNum 20 snap 0 seq_num 20) 2014-08-22 19:27:10,778.778 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:10,778.778 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:10,779.779 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:27:10,779.779 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:27:11,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.160316 7fd732e90700 1 librados: starting msgr at :/0 2014-08-22 19:27:11,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.160328 7fd732e90700 1 librados: starting objecter 2014-08-22 19:27:11,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.160600 7fd732e90700 1 librados: setting wanted keys 2014-08-22 19:27:11,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.160606 7fd732e90700 1 librados: calling monclient init 2014-08-22 19:27:11,256.256 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.162141 7fd72b7fe700 10 client.?.objecter ms_handle_connect 0x7fd72c0355e0 2014-08-22 19:27:11,256.256 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.162184 7fd72b7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:11,258.258 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.165029 7fd732e90700 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:11,259.259 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.165216 7fd732e90700 1 librados: init done 2014-08-22 19:27:11,260.260 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.165781 7fd72b7fe700 10 client.4199.objecter ms_dispatch 0x7fd72c000f80 osd_map(146..146 src has 1..146) v3 2014-08-22 19:27:11,260.260 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.165802 7fd72b7fe700 3 client.4199.objecter handle_osd_map got epochs [146,146] > 0 2014-08-22 19:27:11,260.260 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.165808 7fd72b7fe700 3 client.4199.objecter handle_osd_map decoding full epoch 146 2014-08-22 19:27:11,260.260 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.166021 7fd72b7fe700 20 client.4199.objecter dump_active .. 0 homeless 2014-08-22 19:27:11,261.261 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.166119 7fd72b7fe700 10 client.4199.objecter ms_dispatch 0x7fd72c000f80 osd_map(146..146 src has 1..146) v3 2014-08-22 19:27:11,329.329 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.166133 7fd72b7fe700 3 client.4199.objecter handle_osd_map ignoring epochs [146,146] <= 146 2014-08-22 19:27:11,329.329 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.166139 7fd72b7fe700 20 client.4199.objecter dump_active .. 0 homeless 2014-08-22 19:27:11,349.349 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:27:11,625.625 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.262866 7fd732e90700 20 client.4199.objecter shutdown clearing up homeless session... 2014-08-22 19:27:11,625.625 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:04.263949 7fd732e90700 1 librados: shutdown 2014-08-22 19:27:11,803.803 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:27:11,803.803 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 74' 2014-08-22 19:27:12,022.022 INFO:tasks.rados.rados.0.mira097.stdout:983: done (5 left) 2014-08-22 19:27:12,022.022 INFO:tasks.rados.rados.0.mira097.stdout:986: done (4 left) 2014-08-22 19:27:12,022.022 INFO:tasks.rados.rados.0.mira097.stdout:988: done (3 left) 2014-08-22 19:27:12,027.027 INFO:tasks.rados.rados.0.mira097.stdout:989: done (2 left) 2014-08-22 19:27:12,027.027 INFO:tasks.rados.rados.0.mira097.stdout:990: done (1 left) 2014-08-22 19:27:12,027.027 INFO:tasks.rados.rados.0.mira097.stdout:991: write oid 457 current snap is 51 2014-08-22 19:27:12,027.027 INFO:tasks.rados.rados.0.mira097.stdout:991: seq_num 681 ranges {740351=400000} 2014-08-22 19:27:12,050.050 INFO:tasks.rados.rados.0.mira097.stdout:991: writing mira09725123-457 from 740351 to 1140351 tid 1 2014-08-22 19:27:12,052.052 INFO:tasks.rados.rados.0.mira097.stdout:992: copy_from oid 48 from oid 434 current snap is 51 2014-08-22 19:27:12,052.052 INFO:tasks.rados.rados.0.mira097.stdout:993: read oid 214 snap -1 2014-08-22 19:27:12,052.052 INFO:tasks.rados.rados.0.mira097.stdout:994: delete oid 241 current snap is 51 2014-08-22 19:27:12,166.166 INFO:tasks.rados.rados.0.mira097.stdout:981: expect (ObjNum 335 snap 0 seq_num 335) 2014-08-22 19:27:12,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.337958 7f6310c6f700 1 librados: starting msgr at :/0 2014-08-22 19:27:12,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.337964 7f6310c6f700 1 librados: starting objecter 2014-08-22 19:27:12,439.439 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.338140 7f6310c6f700 1 librados: setting wanted keys 2014-08-22 19:27:12,439.439 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.338147 7f6310c6f700 1 librados: calling monclient init 2014-08-22 19:27:12,439.439 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.338875 7f6309ae7700 10 client.?.objecter ms_handle_connect 0x7f630c0355e0 2014-08-22 19:27:12,439.439 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.338895 7f6309ae7700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:12,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340339 7f6310c6f700 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:12,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340447 7f6310c6f700 1 librados: init done 2014-08-22 19:27:12,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340758 7f6309ae7700 10 client.4215.objecter ms_dispatch 0x7f630c000f80 osd_map(146..146 src has 1..146) v3 2014-08-22 19:27:12,440.440 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340767 7f6309ae7700 3 client.4215.objecter handle_osd_map got epochs [146,146] > 0 2014-08-22 19:27:12,441.441 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340771 7f6309ae7700 3 client.4215.objecter handle_osd_map decoding full epoch 146 2014-08-22 19:27:12,441.441 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340882 7f6309ae7700 20 client.4215.objecter dump_active .. 0 homeless 2014-08-22 19:27:12,441.441 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340930 7f6309ae7700 10 client.4215.objecter ms_dispatch 0x7f630c000f80 osd_map(146..146 src has 1..146) v3 2014-08-22 19:27:12,441.441 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340934 7f6309ae7700 3 client.4215.objecter handle_osd_map ignoring epochs [146,146] <= 146 2014-08-22 19:27:12,441.441 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.340937 7f6309ae7700 20 client.4215.objecter dump_active .. 0 homeless 2014-08-22 19:27:13,083.083 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 74 2014-08-22 19:27:13,089.089 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.994440 7f6310c6f700 20 client.4215.objecter shutdown clearing up homeless session... 2014-08-22 19:27:13,089.089 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:05.994878 7f6310c6f700 1 librados: shutdown 2014-08-22 19:27:13,902.902 INFO:tasks.rados.rados.0.mira097.stdout:991: finishing write tid 1 to mira09725123-457 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:991: finishing write tid 3 to mira09725123-457 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:991: finishing write tid 4 to mira09725123-457 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 457 v 1484 (ObjNum 681 snap 51 seq_num 681) dirty exists 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:981: done (4 left) 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:991: done (3 left) 2014-08-22 19:27:13,903.903 INFO:tasks.rados.rados.0.mira097.stdout:994: done (2 left) 2014-08-22 19:27:13,904.904 INFO:tasks.rados.rados.0.mira097.stdout:995: snap_remove snap 49 2014-08-22 19:27:13,904.904 INFO:tasks.rados.rados.0.mira097.stdout:992: finishing copy_from to mira09725123-48 2014-08-22 19:27:13,904.904 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 48 v 660 (ObjNum 433 snap 0 seq_num 433) dirty exists 2014-08-22 19:27:13,904.904 INFO:tasks.rados.rados.0.mira097.stdout:993: expect (ObjNum 213 snap 0 seq_num 213) 2014-08-22 19:27:13,911.911 INFO:tasks.rados.rados.0.mira097.stdout:992: finishing copy_from racing read to mira09725123-48 2014-08-22 19:27:14,972.972 INFO:tasks.rados.rados.0.mira097.stdout:992: done (2 left) 2014-08-22 19:27:14,972.972 INFO:tasks.rados.rados.0.mira097.stdout:993: done (1 left) 2014-08-22 19:27:14,972.972 INFO:tasks.rados.rados.0.mira097.stdout:995: done (0 left) 2014-08-22 19:27:14,972.972 INFO:tasks.rados.rados.0.mira097.stdout:996: delete oid 289 current snap is 51 2014-08-22 19:27:14,987.987 INFO:tasks.rados.rados.0.mira097.stdout:996: done (0 left) 2014-08-22 19:27:14,987.987 INFO:tasks.rados.rados.0.mira097.stdout:997: snap_remove snap 33 2014-08-22 19:27:16,601.601 INFO:tasks.rados.rados.0.mira097.stdout:997: done (0 left) 2014-08-22 19:27:16,601.601 INFO:tasks.rados.rados.0.mira097.stdout:998: read oid 201 snap 45 2014-08-22 19:27:16,601.601 INFO:tasks.rados.rados.0.mira097.stdout:999: copy_from oid 404 from oid 492 current snap is 51 2014-08-22 19:27:16,602.602 INFO:tasks.rados.rados.0.mira097.stdout:1000: delete oid 295 current snap is 51 2014-08-22 19:27:16,658.658 INFO:tasks.rados.rados.0.mira097.stdout:998: expect (ObjNum 200 snap 0 seq_num 200) 2014-08-22 19:27:17,519.519 INFO:tasks.rados.rados.0.mira097.stdout:999: finishing copy_from to mira09725123-404 2014-08-22 19:27:17,519.519 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 404 v 611 (ObjNum 491 snap 0 seq_num 491) dirty exists 2014-08-22 19:27:17,519.519 INFO:tasks.rados.rados.0.mira097.stdout:999: finishing copy_from racing read to mira09725123-404 2014-08-22 19:27:17,520.520 INFO:tasks.rados.rados.0.mira097.stdout:998: done (2 left) 2014-08-22 19:27:17,520.520 INFO:tasks.rados.rados.0.mira097.stdout:999: done (1 left) 2014-08-22 19:27:17,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1000: done (0 left) 2014-08-22 19:27:17,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1001: snap_remove snap 32 2014-08-22 19:27:18,094.094 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:18,094.094 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:18,095.095 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:27:18,095.095 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:27:18,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.469457 7f6d44bc7700 1 librados: starting msgr at :/0 2014-08-22 19:27:18,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.469469 7f6d44bc7700 1 librados: starting objecter 2014-08-22 19:27:18,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.469643 7f6d44bc7700 1 librados: setting wanted keys 2014-08-22 19:27:18,564.564 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.469650 7f6d44bc7700 1 librados: calling monclient init 2014-08-22 19:27:18,565.565 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.470439 7f6d3dae7700 10 client.?.objecter ms_handle_connect 0x7f6d400355e0 2014-08-22 19:27:18,565.565 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.470463 7f6d3dae7700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:18,590.590 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495199 7f6d44bc7700 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:18,590.590 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495363 7f6d44bc7700 1 librados: init done 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495555 7f6d3dae7700 10 client.4218.objecter ms_dispatch 0x7f6d40000f80 osd_map(149..149 src has 1..149) v3 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495565 7f6d3dae7700 3 client.4218.objecter handle_osd_map got epochs [149,149] > 0 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495569 7f6d3dae7700 3 client.4218.objecter handle_osd_map decoding full epoch 149 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495658 7f6d3dae7700 20 client.4218.objecter dump_active .. 0 homeless 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495704 7f6d3dae7700 10 client.4218.objecter ms_dispatch 0x7f6d40000f80 osd_map(149..149 src has 1..149) v3 2014-08-22 19:27:18,591.591 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495710 7f6d3dae7700 3 client.4218.objecter handle_osd_map ignoring epochs [149,149] <= 149 2014-08-22 19:27:18,592.592 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:11.495713 7f6d3dae7700 20 client.4218.objecter dump_active .. 0 homeless 2014-08-22 19:27:19,635.635 INFO:tasks.rados.rados.0.mira097.stdout:1001: done (0 left) 2014-08-22 19:27:19,635.635 INFO:tasks.rados.rados.0.mira097.stdout:1002: rollback oid 106 current snap is 51 2014-08-22 19:27:19,635.635 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 106 to 48 2014-08-22 19:27:19,636.636 INFO:tasks.rados.rados.0.mira097.stdout:1003: read oid 239 snap -1 2014-08-22 19:27:19,636.636 INFO:tasks.rados.rados.0.mira097.stdout:1004: delete oid 265 current snap is 51 2014-08-22 19:27:19,673.673 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:27:20,034.034 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 106 v 1470 (ObjNum 658 snap 46 seq_num 658) dirty exists 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1002: done (2 left) 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1004: done (1 left) 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1005: copy_from oid 345 from oid 398 current snap is 51 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1006: read oid 15 snap -1 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1007: copy_from oid 47 from oid 442 current snap is 51 2014-08-22 19:27:20,035.035 INFO:tasks.rados.rados.0.mira097.stdout:1008: delete oid 410 current snap is 51 2014-08-22 19:27:20,036.036 INFO:tasks.rados.rados.0.mira097.stdout:1003: expect (ObjNum 677 snap 51 seq_num 677) 2014-08-22 19:27:20,036.036 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:12.585459 7f6d44bc7700 20 client.4218.objecter shutdown clearing up homeless session... 2014-08-22 19:27:20,036.036 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:12.585807 7f6d44bc7700 1 librados: shutdown 2014-08-22 19:27:21,733.733 INFO:tasks.rados.rados.0.mira097.stdout:1005: finishing copy_from to mira09725123-345 2014-08-22 19:27:21,734.734 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 345 v 475 (ObjNum 397 snap 0 seq_num 397) dirty exists 2014-08-22 19:27:21,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1006: expect (ObjNum 14 snap 0 seq_num 14) 2014-08-22 19:27:22,742.742 INFO:tasks.rados.rados.0.mira097.stdout:1007: finishing copy_from to mira09725123-47 2014-08-22 19:27:22,742.742 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 47 v 724 (ObjNum 441 snap 0 seq_num 441) dirty exists 2014-08-22 19:27:22,742.742 INFO:tasks.rados.rados.0.mira097.stdout:1005: finishing copy_from racing read to mira09725123-345 2014-08-22 19:27:22,742.742 INFO:tasks.rados.rados.0.mira097.stdout:1007: finishing copy_from racing read to mira09725123-47 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1003: done (4 left) 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1005: done (3 left) 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1006: done (2 left) 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1007: done (1 left) 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1008: done (0 left) 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1009: copy_from oid 99 from oid 372 current snap is 51 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1010: read oid 199 snap 47 2014-08-22 19:27:22,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1011: write oid 16 current snap is 51 2014-08-22 19:27:22,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1011: seq_num 687 ranges {25456=1} 2014-08-22 19:27:22,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1011: writing mira09725123-16 from 25456 to 25457 tid 1 2014-08-22 19:27:22,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1012: write oid 129 current snap is 51 2014-08-22 19:27:22,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1012: seq_num 688 ranges {521711=453451,1760029=642238,3090369=612099,3721710=1} 2014-08-22 19:27:22,919.919 INFO:tasks.rados.rados.0.mira097.stdout:1012: writing mira09725123-129 from 521711 to 975162 tid 1 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1012: writing mira09725123-129 from 1760029 to 2402267 tid 2 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1012: writing mira09725123-129 from 3090369 to 3702468 tid 3 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1012: writing mira09725123-129 from 3721710 to 3721711 tid 4 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1013: copy_from oid 330 from oid 453 current snap is 51 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1009: finishing copy_from to mira09725123-99 2014-08-22 19:27:22,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1009: got expected ENOENT (src dne) 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1010: done (4 left) 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1014: write oid 11 current snap is 51 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1014: seq_num 689 ranges {714093=426279,1514092=1} 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1014: writing mira09725123-11 from 714093 to 1140372 tid 1 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1014: writing mira09725123-11 from 1514092 to 1514093 tid 2 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1015: write oid 292 current snap is 51 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1015: seq_num 690 ranges {526698=549508,1518149=208549} 2014-08-22 19:27:22,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1015: writing mira09725123-292 from 526698 to 1076206 tid 1 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1015: writing mira09725123-292 from 1518149 to 1726698 tid 2 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1016: read oid 165 snap -1 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1017: write oid 113 current snap is 51 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1017: seq_num 691 ranges {621928=521312,1421927=1} 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1017: writing mira09725123-113 from 621928 to 1143240 tid 1 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1017: writing mira09725123-113 from 1421927 to 1421928 tid 2 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1018: copy_from oid 84 from oid 218 current snap is 51 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1019: read oid 342 snap -1 2014-08-22 19:27:22,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1020: snap_create 2014-08-22 19:27:22,923.923 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 10 2014-08-22 19:27:22,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1016: expect (ObjNum 534 snap 8 seq_num 534) 2014-08-22 19:27:23,359.359 INFO:tasks.rados.rados.0.mira097.stdout:1019: expect (ObjNum 115 snap 0 seq_num 115) 2014-08-22 19:27:23,398.398 INFO:tasks.rados.rados.0.mira097.stdout:1009: finishing copy_from racing read to mira09725123-99 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 1 to mira09725123-129 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1011: finishing write tid 1 to mira09725123-16 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1011: finishing write tid 3 to mira09725123-16 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 2 to mira09725123-129 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 3 to mira09725123-129 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1011: finishing write tid 4 to mira09725123-16 2014-08-22 19:27:23,399.399 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 16 v 1485 (ObjNum 687 snap 51 seq_num 687) dirty exists 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 4 to mira09725123-129 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1009: done (9 left) 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1011: done (8 left) 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1016: done (7 left) 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1019: done (6 left) 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 6 to mira09725123-129 2014-08-22 19:27:23,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1014: finishing write tid 1 to mira09725123-11 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1014: finishing write tid 2 to mira09725123-11 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1014: finishing write tid 4 to mira09725123-11 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1012: finishing write tid 7 to mira09725123-129 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 129 v 1523 (ObjNum 688 snap 51 seq_num 688) dirty exists 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1014: finishing write tid 5 to mira09725123-11 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 11 v 1490 (ObjNum 689 snap 51 seq_num 689) dirty exists 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1015: finishing write tid 1 to mira09725123-292 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1015: finishing write tid 2 to mira09725123-292 2014-08-22 19:27:23,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1015: finishing write tid 4 to mira09725123-292 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1018: finishing copy_from to mira09725123-84 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 84 v 1382 (ObjNum 217 snap 0 seq_num 217) dirty exists 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1017: finishing write tid 1 to mira09725123-113 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1017: finishing write tid 2 to mira09725123-113 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1018: finishing copy_from racing read to mira09725123-84 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1017: finishing write tid 4 to mira09725123-113 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1013: finishing copy_from to mira09725123-330 2014-08-22 19:27:23,402.402 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 330 v 544 (ObjNum 452 snap 0 seq_num 452) dirty exists 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1012: done (5 left) 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1014: done (4 left) 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1018: done (3 left) 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1017: finishing write tid 5 to mira09725123-113 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 113 v 1499 (ObjNum 691 snap 51 seq_num 691) dirty exists 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1015: finishing write tid 5 to mira09725123-292 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 292 v 1474 (ObjNum 690 snap 51 seq_num 690) dirty exists 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1013: finishing copy_from racing read to mira09725123-330 2014-08-22 19:27:23,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1013: done (2 left) 2014-08-22 19:27:23,404.404 INFO:tasks.rados.rados.0.mira097.stdout:1015: done (1 left) 2014-08-22 19:27:23,404.404 INFO:tasks.rados.rados.0.mira097.stdout:1017: done (0 left) 2014-08-22 19:27:25,219.219 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:25,219.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:25,219.219 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:27:25,220.220 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:27:25,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1020: done (0 left) 2014-08-22 19:27:25,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1021: rollback oid 361 current snap is 52 2014-08-22 19:27:25,454.454 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 361 to 51 2014-08-22 19:27:25,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1022: read oid 54 snap -1 2014-08-22 19:27:25,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1023: read oid 289 snap -1 2014-08-22 19:27:25,455.455 INFO:tasks.rados.rados.0.mira097.stdout:1024: snap_remove snap 48 2014-08-22 19:27:25,489.489 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 361 v 1501 (ObjNum 650 snap 43 seq_num 650) dirty exists 2014-08-22 19:27:25,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1022: expect (ObjNum 53 snap 0 seq_num 53) 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.573585 7f322ae82700 1 librados: starting msgr at :/0 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.573592 7f322ae82700 1 librados: starting objecter 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.573717 7f322ae82700 1 librados: setting wanted keys 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.573724 7f322ae82700 1 librados: calling monclient init 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.574562 7f321b7fe700 10 client.?.objecter ms_handle_connect 0x7f32240355e0 2014-08-22 19:27:25,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.574586 7f321b7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:25,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576021 7f322ae82700 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:25,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576116 7f322ae82700 1 librados: init done 2014-08-22 19:27:25,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576441 7f321b7fe700 10 client.4202.objecter ms_dispatch 0x7f3224000f80 osd_map(151..151 src has 1..151) v3 2014-08-22 19:27:25,683.683 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576451 7f321b7fe700 3 client.4202.objecter handle_osd_map got epochs [151,151] > 0 2014-08-22 19:27:25,684.684 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576454 7f321b7fe700 3 client.4202.objecter handle_osd_map decoding full epoch 151 2014-08-22 19:27:25,684.684 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576551 7f321b7fe700 20 client.4202.objecter dump_active .. 0 homeless 2014-08-22 19:27:25,684.684 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576589 7f321b7fe700 10 client.4202.objecter ms_dispatch 0x7f3224000f80 osd_map(151..151 src has 1..151) v3 2014-08-22 19:27:25,684.684 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576595 7f321b7fe700 3 client.4202.objecter handle_osd_map ignoring epochs [151,151] <= 151 2014-08-22 19:27:25,685.685 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.576598 7f321b7fe700 20 client.4202.objecter dump_active .. 0 homeless 2014-08-22 19:27:25,850.850 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:27:26,117.117 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.666007 7f322ae82700 20 client.4202.objecter shutdown clearing up homeless session... 2014-08-22 19:27:26,118.118 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:18.666511 7f322ae82700 1 librados: shutdown 2014-08-22 19:27:26,301.301 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 74' 2014-08-22 19:27:26,461.461 INFO:tasks.rados.rados.0.mira097.stdout:1021: done (3 left) 2014-08-22 19:27:26,462.462 INFO:tasks.rados.rados.0.mira097.stdout:1022: done (2 left) 2014-08-22 19:27:26,462.462 INFO:tasks.rados.rados.0.mira097.stdout:1023: done (1 left) 2014-08-22 19:27:26,462.462 INFO:tasks.rados.rados.0.mira097.stdout:1024: done (0 left) 2014-08-22 19:27:26,462.462 INFO:tasks.rados.rados.0.mira097.stdout:1025: rollback oid 366 current snap is 52 2014-08-22 19:27:26,463.463 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 366 to 42 2014-08-22 19:27:26,463.463 INFO:tasks.rados.rados.0.mira097.stdout:1026: write oid 163 current snap is 52 2014-08-22 19:27:26,463.463 INFO:tasks.rados.rados.0.mira097.stdout:1026: seq_num 692 ranges {415334=528188,1592941=521995,2887526=327808} 2014-08-22 19:27:26,489.489 INFO:tasks.rados.rados.0.mira097.stdout:1026: writing mira09725123-163 from 415334 to 943522 tid 1 2014-08-22 19:27:26,512.512 INFO:tasks.rados.rados.0.mira097.stdout:1026: writing mira09725123-163 from 1592941 to 2114936 tid 2 2014-08-22 19:27:26,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1026: writing mira09725123-163 from 2887526 to 3215334 tid 3 2014-08-22 19:27:26,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1027: write oid 454 current snap is 52 2014-08-22 19:27:26,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1027: seq_num 693 ranges {511587=501017,1556673=743614,2911586=1} 2014-08-22 19:27:26,547.547 INFO:tasks.rados.rados.0.mira097.stdout:1027: writing mira09725123-454 from 511587 to 1012604 tid 1 2014-08-22 19:27:26,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1027: writing mira09725123-454 from 1556673 to 2300287 tid 2 2014-08-22 19:27:26,580.580 INFO:tasks.rados.rados.0.mira097.stdout:1027: writing mira09725123-454 from 2911586 to 2911587 tid 3 2014-08-22 19:27:26,581.581 INFO:tasks.rados.rados.0.mira097.stdout:1028: copy_from oid 267 from oid 195 current snap is 52 2014-08-22 19:27:26,581.581 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 366 v 1515 (ObjNum 365 snap 0 seq_num 365) dirty exists 2014-08-22 19:27:26,581.581 INFO:tasks.rados.rados.0.mira097.stdout:1025: done (3 left) 2014-08-22 19:27:26,581.581 INFO:tasks.rados.rados.0.mira097.stdout:1029: write oid 95 current snap is 52 2014-08-22 19:27:26,581.581 INFO:tasks.rados.rados.0.mira097.stdout:1029: seq_num 694 ranges {606817=624245,1937557=448881,2606816=1} 2014-08-22 19:27:26,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1029: writing mira09725123-95 from 606817 to 1231062 tid 1 2014-08-22 19:27:26,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1029: writing mira09725123-95 from 1937557 to 2386438 tid 2 2014-08-22 19:27:26,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1029: writing mira09725123-95 from 2606816 to 2606817 tid 3 2014-08-22 19:27:26,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1030: snap_remove snap 42 2014-08-22 19:27:26,902.902 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.806836 7ffe66dcb700 1 librados: starting msgr at :/0 2014-08-22 19:27:26,902.902 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.806841 7ffe66dcb700 1 librados: starting objecter 2014-08-22 19:27:26,902.902 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.807025 7ffe66dcb700 1 librados: setting wanted keys 2014-08-22 19:27:26,903.903 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.807030 7ffe66dcb700 1 librados: calling monclient init 2014-08-22 19:27:26,903.903 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.807953 7ffe577fe700 10 client.?.objecter ms_handle_connect 0x7ffe600355e0 2014-08-22 19:27:26,903.903 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.807997 7ffe577fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:26,904.904 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.810542 7ffe66dcb700 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:26,904.904 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.810718 7ffe66dcb700 1 librados: init done 2014-08-22 19:27:26,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.810941 7ffe577fe700 10 client.4205.objecter ms_dispatch 0x7ffe60000f80 osd_map(152..152 src has 1..152) v3 2014-08-22 19:27:26,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.810949 7ffe577fe700 3 client.4205.objecter handle_osd_map got epochs [152,152] > 0 2014-08-22 19:27:26,905.905 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.810951 7ffe577fe700 3 client.4205.objecter handle_osd_map decoding full epoch 152 2014-08-22 19:27:26,906.906 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.811047 7ffe577fe700 20 client.4205.objecter dump_active .. 0 homeless 2014-08-22 19:27:26,906.906 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.811079 7ffe577fe700 10 client.4205.objecter ms_dispatch 0x7ffe60000f80 osd_map(152..152 src has 1..152) v3 2014-08-22 19:27:26,907.907 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.811082 7ffe577fe700 3 client.4205.objecter handle_osd_map ignoring epochs [152,152] <= 152 2014-08-22 19:27:26,907.907 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:19.811083 7ffe577fe700 20 client.4205.objecter dump_active .. 0 homeless 2014-08-22 19:27:27,108.108 INFO:tasks.rados.rados.0.mira097.stdout:1026: finishing write tid 1 to mira09725123-163 2014-08-22 19:27:27,108.108 INFO:tasks.rados.rados.0.mira097.stdout:1027: finishing write tid 1 to mira09725123-454 2014-08-22 19:27:27,108.108 INFO:tasks.rados.rados.0.mira097.stdout:1027: finishing write tid 2 to mira09725123-454 2014-08-22 19:27:27,109.109 INFO:tasks.rados.rados.0.mira097.stdout:1026: finishing write tid 2 to mira09725123-163 2014-08-22 19:27:27,109.109 INFO:tasks.rados.rados.0.mira097.stdout:1026: finishing write tid 3 to mira09725123-163 2014-08-22 19:27:27,111.111 INFO:tasks.rados.rados.0.mira097.stdout:1026: finishing write tid 5 to mira09725123-163 2014-08-22 19:27:27,140.140 INFO:tasks.rados.rados.0.mira097.stdout:1027: finishing write tid 3 to mira09725123-454 2014-08-22 19:27:27,141.141 INFO:tasks.rados.rados.0.mira097.stdout:1027: finishing write tid 5 to mira09725123-454 2014-08-22 19:27:27,146.146 INFO:tasks.rados.rados.0.mira097.stdout:1027: finishing write tid 6 to mira09725123-454 2014-08-22 19:27:27,146.146 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 454 v 1545 (ObjNum 693 snap 52 seq_num 693) dirty exists 2014-08-22 19:27:27,148.148 INFO:tasks.rados.rados.0.mira097.stdout:1029: finishing write tid 1 to mira09725123-95 2014-08-22 19:27:27,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1029: finishing write tid 2 to mira09725123-95 2014-08-22 19:27:27,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1029: finishing write tid 3 to mira09725123-95 2014-08-22 19:27:27,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1029: finishing write tid 5 to mira09725123-95 2014-08-22 19:27:27,156.156 INFO:tasks.rados.rados.0.mira097.stdout:1028: finishing copy_from to mira09725123-267 2014-08-22 19:27:27,156.156 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 267 v 629 (ObjNum 502 snap 2 seq_num 502) dirty exists 2014-08-22 19:27:27,168.168 INFO:tasks.rados.rados.0.mira097.stdout:1026: finishing write tid 6 to mira09725123-163 2014-08-22 19:27:27,168.168 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 163 v 1521 (ObjNum 692 snap 52 seq_num 692) dirty exists 2014-08-22 19:27:27,202.202 INFO:tasks.rados.rados.0.mira097.stdout:1029: finishing write tid 6 to mira09725123-95 2014-08-22 19:27:27,203.203 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 95 v 1531 (ObjNum 694 snap 52 seq_num 694) dirty exists 2014-08-22 19:27:27,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1028: finishing copy_from racing read to mira09725123-267 2014-08-22 19:27:27,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1026: done (4 left) 2014-08-22 19:27:27,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1027: done (3 left) 2014-08-22 19:27:27,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1028: done (2 left) 2014-08-22 19:27:27,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1029: done (1 left) 2014-08-22 19:27:27,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1030: done (0 left) 2014-08-22 19:27:27,503.503 INFO:tasks.rados.rados.0.mira097.stdout:1031: read oid 282 snap 47 2014-08-22 19:27:27,503.503 INFO:tasks.rados.rados.0.mira097.stdout:1032: delete oid 326 current snap is 52 2014-08-22 19:27:27,517.517 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 74 2014-08-22 19:27:27,517.517 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:20.422844 7ffe66dcb700 20 client.4205.objecter shutdown clearing up homeless session... 2014-08-22 19:27:27,517.517 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:20.423368 7ffe66dcb700 1 librados: shutdown 2014-08-22 19:27:27,560.560 INFO:tasks.rados.rados.0.mira097.stdout:1031: expect (ObjNum 281 snap 0 seq_num 281) 2014-08-22 19:27:27,905.905 INFO:tasks.rados.rados.0.mira097.stdout:1031: done (1 left) 2014-08-22 19:27:27,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1032: done (0 left) 2014-08-22 19:27:27,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1033: copy_from oid 300 from oid 220 current snap is 52 2014-08-22 19:27:27,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1034: write oid 82 current snap is 52 2014-08-22 19:27:27,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1034: seq_num 696 ranges {610780=1} 2014-08-22 19:27:27,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1034: writing mira09725123-82 from 610780 to 610781 tid 1 2014-08-22 19:27:27,910.910 INFO:tasks.rados.rados.0.mira097.stdout:1035: delete oid 328 current snap is 52 2014-08-22 19:27:27,920.920 INFO:tasks.rados.rados.0.mira097.stdout:1034: finishing write tid 1 to mira09725123-82 2014-08-22 19:27:27,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1034: finishing write tid 3 to mira09725123-82 2014-08-22 19:27:27,921.921 INFO:tasks.rados.rados.0.mira097.stdout:1034: finishing write tid 4 to mira09725123-82 2014-08-22 19:27:27,922.922 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 82 v 1557 (ObjNum 696 snap 52 seq_num 696) dirty exists 2014-08-22 19:27:27,928.928 INFO:tasks.rados.rados.0.mira097.stdout:1034: done (2 left) 2014-08-22 19:27:27,928.928 INFO:tasks.rados.rados.0.mira097.stdout:1035: done (1 left) 2014-08-22 19:27:27,928.928 INFO:tasks.rados.rados.0.mira097.stdout:1036: snap_remove snap 46 2014-08-22 19:27:28,044.044 INFO:tasks.rados.rados.0.mira097.stdout:1033: finishing copy_from to mira09725123-300 2014-08-22 19:27:28,044.044 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 300 v 263 (ObjNum 219 snap 0 seq_num 219) dirty exists 2014-08-22 19:27:28,044.044 INFO:tasks.rados.rados.0.mira097.stdout:1033: finishing copy_from racing read to mira09725123-300 2014-08-22 19:27:28,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1033: done (1 left) 2014-08-22 19:27:28,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1036: done (0 left) 2014-08-22 19:27:28,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1037: write oid 325 current snap is 52 2014-08-22 19:27:28,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1037: seq_num 698 ranges {500440=547558,1503717=481086,2100439=1} 2014-08-22 19:27:28,545.545 INFO:tasks.rados.rados.0.mira097.stdout:1037: writing mira09725123-325 from 500440 to 1047998 tid 1 2014-08-22 19:27:28,571.571 INFO:tasks.rados.rados.0.mira097.stdout:1037: writing mira09725123-325 from 1503717 to 1984803 tid 2 2014-08-22 19:27:28,572.572 INFO:tasks.rados.rados.0.mira097.stdout:1037: writing mira09725123-325 from 2100439 to 2100440 tid 3 2014-08-22 19:27:28,573.573 INFO:tasks.rados.rados.0.mira097.stdout:1038: write oid 222 current snap is 52 2014-08-22 19:27:28,573.573 INFO:tasks.rados.rados.0.mira097.stdout:1038: seq_num 699 ranges {712022=519363,1946682=365340} 2014-08-22 19:27:28,593.593 INFO:tasks.rados.rados.0.mira097.stdout:1038: writing mira09725123-222 from 712022 to 1231385 tid 1 2014-08-22 19:27:28,610.610 INFO:tasks.rados.rados.0.mira097.stdout:1038: writing mira09725123-222 from 1946682 to 2312022 tid 2 2014-08-22 19:27:28,611.611 INFO:tasks.rados.rados.0.mira097.stdout:1039: copy_from oid 120 from oid 31 current snap is 52 2014-08-22 19:27:28,611.611 INFO:tasks.rados.rados.0.mira097.stdout:1037: finishing write tid 1 to mira09725123-325 2014-08-22 19:27:28,612.612 INFO:tasks.rados.rados.0.mira097.stdout:1037: finishing write tid 2 to mira09725123-325 2014-08-22 19:27:28,612.612 INFO:tasks.rados.rados.0.mira097.stdout:1037: finishing write tid 3 to mira09725123-325 2014-08-22 19:27:28,613.613 INFO:tasks.rados.rados.0.mira097.stdout:1037: finishing write tid 5 to mira09725123-325 2014-08-22 19:27:28,613.613 INFO:tasks.rados.rados.0.mira097.stdout:1037: finishing write tid 6 to mira09725123-325 2014-08-22 19:27:28,613.613 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 325 v 1608 (ObjNum 698 snap 52 seq_num 698) dirty exists 2014-08-22 19:27:28,613.613 INFO:tasks.rados.rados.0.mira097.stdout:1037: done (2 left) 2014-08-22 19:27:28,613.613 INFO:tasks.rados.rados.0.mira097.stdout:1040: rollback oid 481 current snap is 52 2014-08-22 19:27:28,617.617 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 481 to 43 2014-08-22 19:27:28,617.617 INFO:tasks.rados.rados.0.mira097.stdout:1041: copy_from oid 26 from oid 236 current snap is 52 2014-08-22 19:27:28,617.617 INFO:tasks.rados.rados.0.mira097.stdout:1042: write oid 149 current snap is 52 2014-08-22 19:27:28,617.617 INFO:tasks.rados.rados.0.mira097.stdout:1042: seq_num 700 ranges {408276=643615,1812238=196038} 2014-08-22 19:27:28,633.633 INFO:tasks.rados.rados.0.mira097.stdout:1042: writing mira09725123-149 from 408276 to 1051891 tid 1 2014-08-22 19:27:28,647.647 INFO:tasks.rados.rados.0.mira097.stdout:1042: writing mira09725123-149 from 1812238 to 2008276 tid 2 2014-08-22 19:27:28,648.648 INFO:tasks.rados.rados.0.mira097.stdout:1043: write oid 206 current snap is 52 2014-08-22 19:27:28,648.648 INFO:tasks.rados.rados.0.mira097.stdout:1043: seq_num 701 ranges {752081=499068,1657570=443095,2752080=1} 2014-08-22 19:27:28,668.668 INFO:tasks.rados.rados.0.mira097.stdout:1043: writing mira09725123-206 from 752081 to 1251149 tid 1 2014-08-22 19:27:28,684.684 INFO:tasks.rados.rados.0.mira097.stdout:1043: writing mira09725123-206 from 1657570 to 2100665 tid 2 2014-08-22 19:27:28,689.689 INFO:tasks.rados.rados.0.mira097.stdout:1043: writing mira09725123-206 from 2752080 to 2752081 tid 3 2014-08-22 19:27:28,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1044: snap_remove snap 51 2014-08-22 19:27:28,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1042: finishing write tid 1 to mira09725123-149 2014-08-22 19:27:28,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1042: finishing write tid 2 to mira09725123-149 2014-08-22 19:27:28,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1042: finishing write tid 4 to mira09725123-149 2014-08-22 19:27:28,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1042: finishing write tid 5 to mira09725123-149 2014-08-22 19:27:28,691.691 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 149 v 1382 (ObjNum 700 snap 52 seq_num 700) dirty exists 2014-08-22 19:27:28,691.691 INFO:tasks.rados.rados.0.mira097.stdout:1038: finishing write tid 1 to mira09725123-222 2014-08-22 19:27:28,691.691 INFO:tasks.rados.rados.0.mira097.stdout:1038: finishing write tid 2 to mira09725123-222 2014-08-22 19:27:28,691.691 INFO:tasks.rados.rados.0.mira097.stdout:1038: finishing write tid 4 to mira09725123-222 2014-08-22 19:27:28,748.748 INFO:tasks.rados.rados.0.mira097.stdout:1038: finishing write tid 5 to mira09725123-222 2014-08-22 19:27:28,748.748 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 222 v 1617 (ObjNum 699 snap 52 seq_num 699) dirty exists 2014-08-22 19:27:28,759.759 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 481 v 1620 (ObjNum 480 snap 0 seq_num 480) dirty exists 2014-08-22 19:27:28,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1041: finishing copy_from to mira09725123-26 2014-08-22 19:27:28,792.792 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 26 v 1191 (ObjNum 235 snap 0 seq_num 235) dirty exists 2014-08-22 19:27:28,802.802 INFO:tasks.rados.rados.0.mira097.stdout:1041: finishing copy_from racing read to mira09725123-26 2014-08-22 19:27:28,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1043: finishing write tid 1 to mira09725123-206 2014-08-22 19:27:28,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1043: finishing write tid 2 to mira09725123-206 2014-08-22 19:27:28,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1043: finishing write tid 3 to mira09725123-206 2014-08-22 19:27:28,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1043: finishing write tid 5 to mira09725123-206 2014-08-22 19:27:29,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1043: finishing write tid 6 to mira09725123-206 2014-08-22 19:27:29,115.115 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 206 v 1628 (ObjNum 701 snap 52 seq_num 701) dirty exists 2014-08-22 19:27:29,426.426 INFO:tasks.rados.rados.0.mira097.stdout:1039: finishing copy_from to mira09725123-120 2014-08-22 19:27:29,426.426 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 120 v 32 (ObjNum 30 snap 0 seq_num 30) dirty exists 2014-08-22 19:27:29,427.427 INFO:tasks.rados.rados.0.mira097.stdout:1039: finishing copy_from racing read to mira09725123-120 2014-08-22 19:27:29,518.518 INFO:tasks.rados.rados.0.mira097.stdout:1038: done (6 left) 2014-08-22 19:27:29,519.519 INFO:tasks.rados.rados.0.mira097.stdout:1039: done (5 left) 2014-08-22 19:27:29,519.519 INFO:tasks.rados.rados.0.mira097.stdout:1040: done (4 left) 2014-08-22 19:27:29,519.519 INFO:tasks.rados.rados.0.mira097.stdout:1041: done (3 left) 2014-08-22 19:27:29,519.519 INFO:tasks.rados.rados.0.mira097.stdout:1042: done (2 left) 2014-08-22 19:27:29,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1043: done (1 left) 2014-08-22 19:27:29,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1044: done (0 left) 2014-08-22 19:27:29,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1045: snap_remove snap 39 2014-08-22 19:27:30,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1045: done (0 left) 2014-08-22 19:27:30,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1046: write oid 221 current snap is 52 2014-08-22 19:27:30,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1046: seq_num 702 ranges {696911=622296,1736630=664714,3094083=402828} 2014-08-22 19:27:30,569.569 INFO:tasks.rados.rados.0.mira097.stdout:1046: writing mira09725123-221 from 696911 to 1319207 tid 1 2014-08-22 19:27:30,598.598 INFO:tasks.rados.rados.0.mira097.stdout:1046: writing mira09725123-221 from 1736630 to 2401344 tid 2 2014-08-22 19:27:30,618.618 INFO:tasks.rados.rados.0.mira097.stdout:1046: writing mira09725123-221 from 3094083 to 3496911 tid 3 2014-08-22 19:27:30,618.618 INFO:tasks.rados.rados.0.mira097.stdout:1047: read oid 127 snap -1 2014-08-22 19:27:30,618.618 INFO:tasks.rados.rados.0.mira097.stdout:1048: write oid 256 current snap is 52 2014-08-22 19:27:30,619.619 INFO:tasks.rados.rados.0.mira097.stdout:1048: seq_num 703 ranges {792141=594100,1849739=467134,3027534=164607} 2014-08-22 19:27:30,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1048: writing mira09725123-256 from 792141 to 1386241 tid 1 2014-08-22 19:27:30,661.661 INFO:tasks.rados.rados.0.mira097.stdout:1048: writing mira09725123-256 from 1849739 to 2316873 tid 2 2014-08-22 19:27:30,701.701 INFO:tasks.rados.rados.0.mira097.stdout:1048: writing mira09725123-256 from 3027534 to 3192141 tid 3 2014-08-22 19:27:30,701.701 INFO:tasks.rados.rados.0.mira097.stdout:1049: copy_from oid 430 from oid 491 current snap is 52 2014-08-22 19:27:30,701.701 INFO:tasks.rados.rados.0.mira097.stdout:1050: write oid 129 current snap is 52 2014-08-22 19:27:30,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1050: seq_num 704 ranges {452298=1} 2014-08-22 19:27:30,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1050: writing mira09725123-129 from 452298 to 452299 tid 1 2014-08-22 19:27:30,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1051: read oid 379 snap -1 2014-08-22 19:27:30,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1047: expect (ObjNum 126 snap 0 seq_num 126) 2014-08-22 19:27:30,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1046: finishing write tid 1 to mira09725123-221 2014-08-22 19:27:30,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1046: finishing write tid 2 to mira09725123-221 2014-08-22 19:27:30,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1046: finishing write tid 3 to mira09725123-221 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1046: finishing write tid 5 to mira09725123-221 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1046: finishing write tid 6 to mira09725123-221 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 221 v 1402 (ObjNum 702 snap 52 seq_num 702) dirty exists 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1048: finishing write tid 1 to mira09725123-256 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1046: done (5 left) 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1047: done (4 left) 2014-08-22 19:27:30,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1052: copy_from oid 73 from oid 455 current snap is 52 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1048: finishing write tid 2 to mira09725123-256 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1048: finishing write tid 3 to mira09725123-256 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1048: finishing write tid 5 to mira09725123-256 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1050: finishing write tid 1 to mira09725123-129 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1050: finishing write tid 3 to mira09725123-129 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1050: finishing write tid 4 to mira09725123-129 2014-08-22 19:27:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 129 v 1659 (ObjNum 704 snap 52 seq_num 704) dirty exists 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:1048: finishing write tid 6 to mira09725123-256 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 256 v 1411 (ObjNum 703 snap 52 seq_num 703) dirty exists 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:1049: finishing copy_from to mira09725123-430 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 430 v 604 (ObjNum 490 snap 0 seq_num 490) dirty exists 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:1049: finishing copy_from racing read to mira09725123-430 2014-08-22 19:27:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:1051: expect (ObjNum 378 snap 0 seq_num 378) 2014-08-22 19:27:31,040.040 INFO:tasks.rados.rados.0.mira097.stdout:1048: done (4 left) 2014-08-22 19:27:31,040.040 INFO:tasks.rados.rados.0.mira097.stdout:1049: done (3 left) 2014-08-22 19:27:31,040.040 INFO:tasks.rados.rados.0.mira097.stdout:1050: done (2 left) 2014-08-22 19:27:31,041.041 INFO:tasks.rados.rados.0.mira097.stdout:1051: done (1 left) 2014-08-22 19:27:31,041.041 INFO:tasks.rados.rados.0.mira097.stdout:1053: read oid 427 snap -1 2014-08-22 19:27:31,041.041 INFO:tasks.rados.rados.0.mira097.stdout:1054: snap_remove snap 45 2014-08-22 19:27:31,077.077 INFO:tasks.rados.rados.0.mira097.stdout:1053: expect (ObjNum 426 snap 0 seq_num 426) 2014-08-22 19:27:31,121.121 INFO:tasks.rados.rados.0.mira097.stdout:1052: finishing copy_from to mira09725123-73 2014-08-22 19:27:31,121.121 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 73 v 618 (ObjNum 454 snap 0 seq_num 454) dirty exists 2014-08-22 19:27:31,132.132 INFO:tasks.rados.rados.0.mira097.stdout:1052: finishing copy_from racing read to mira09725123-73 2014-08-22 19:27:31,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1052: done (2 left) 2014-08-22 19:27:31,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1053: done (1 left) 2014-08-22 19:27:31,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1054: done (0 left) 2014-08-22 19:27:31,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1055: write oid 495 current snap is 52 2014-08-22 19:27:31,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1055: seq_num 705 ranges {797128=400000} 2014-08-22 19:27:31,560.560 INFO:tasks.rados.rados.0.mira097.stdout:1055: writing mira09725123-495 from 797128 to 1197128 tid 1 2014-08-22 19:27:31,561.561 INFO:tasks.rados.rados.0.mira097.stdout:1056: write oid 63 current snap is 52 2014-08-22 19:27:31,561.561 INFO:tasks.rados.rados.0.mira097.stdout:1056: seq_num 706 ranges {492358=400000} 2014-08-22 19:27:31,580.580 INFO:tasks.rados.rados.0.mira097.stdout:1056: writing mira09725123-63 from 492358 to 892358 tid 1 2014-08-22 19:27:31,580.580 INFO:tasks.rados.rados.0.mira097.stdout:1057: write oid 310 current snap is 52 2014-08-22 19:27:31,580.580 INFO:tasks.rados.rados.0.mira097.stdout:1057: seq_num 707 ranges {437188=670886,1637187=1} 2014-08-22 19:27:31,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1057: writing mira09725123-310 from 437188 to 1108074 tid 1 2014-08-22 19:27:31,712.712 INFO:tasks.rados.rados.0.mira097.stdout:1057: writing mira09725123-310 from 1637187 to 1637188 tid 2 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1058: rollback oid 118 current snap is 52 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 118 to 43 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1055: finishing write tid 1 to mira09725123-495 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1055: finishing write tid 3 to mira09725123-495 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1055: finishing write tid 4 to mira09725123-495 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 495 v 1576 (ObjNum 705 snap 52 seq_num 705) dirty exists 2014-08-22 19:27:31,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1055: done (3 left) 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1059: rollback oid 66 current snap is 52 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 66 to 50 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1060: write oid 276 current snap is 52 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1060: seq_num 708 ranges {498370=642690,1915184=472382,2800637=97733} 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1060: writing mira09725123-276 from 498370 to 1141060 tid 1 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1060: writing mira09725123-276 from 1915184 to 2387566 tid 2 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1060: writing mira09725123-276 from 2800637 to 2898370 tid 3 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1061: snap_remove snap 47 2014-08-22 19:27:31,714.714 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 66 v 0 (ObjNum 9157280 snap 0 seq_num 0) dirty dne 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1056: finishing write tid 1 to mira09725123-63 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1056: finishing write tid 3 to mira09725123-63 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1060: finishing write tid 1 to mira09725123-276 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1056: finishing write tid 4 to mira09725123-63 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 63 v 1431 (ObjNum 706 snap 52 seq_num 706) dirty exists 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1057: finishing write tid 1 to mira09725123-310 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1057: finishing write tid 2 to mira09725123-310 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1057: finishing write tid 4 to mira09725123-310 2014-08-22 19:27:31,715.715 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 118 v 1620 (ObjNum 117 snap 0 seq_num 117) dirty exists 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1060: finishing write tid 2 to mira09725123-276 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1060: finishing write tid 3 to mira09725123-276 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1060: finishing write tid 5 to mira09725123-276 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1060: finishing write tid 6 to mira09725123-276 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 276 v 1437 (ObjNum 708 snap 52 seq_num 708) dirty exists 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1057: finishing write tid 5 to mira09725123-310 2014-08-22 19:27:31,716.716 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 310 v 1617 (ObjNum 707 snap 52 seq_num 707) dirty exists 2014-08-22 19:27:32,529.529 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:32,529.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:32,529.529 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2014-08-22 19:27:32,530.530 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:27:32,530.530 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:27:32,530.530 INFO:teuthology.orchestra.run.mira097:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-08-22 19:27:32,552.552 INFO:tasks.rados.rados.0.mira097.stdout:1056: done (5 left) 2014-08-22 19:27:32,552.552 INFO:tasks.rados.rados.0.mira097.stdout:1057: done (4 left) 2014-08-22 19:27:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1058: done (3 left) 2014-08-22 19:27:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1059: done (2 left) 2014-08-22 19:27:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1060: done (1 left) 2014-08-22 19:27:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1061: done (0 left) 2014-08-22 19:27:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1062: write oid 389 current snap is 52 2014-08-22 19:27:32,554.554 INFO:tasks.rados.rados.0.mira097.stdout:1062: seq_num 709 ranges {593599=498142,1876867=542577,2593598=1} 2014-08-22 19:27:32,580.580 INFO:tasks.rados.rados.0.mira097.stdout:1062: writing mira09725123-389 from 593599 to 1091741 tid 1 2014-08-22 19:27:32,608.608 INFO:tasks.rados.rados.0.mira097.stdout:1062: writing mira09725123-389 from 1876867 to 2419444 tid 2 2014-08-22 19:27:32,610.610 INFO:tasks.rados.rados.0.mira097.stdout:1062: writing mira09725123-389 from 2593598 to 2593599 tid 3 2014-08-22 19:27:32,611.611 INFO:tasks.rados.rados.0.mira097.stdout:1063: read oid 88 snap -1 2014-08-22 19:27:32,611.611 INFO:tasks.rados.rados.0.mira097.stdout:1064: snap_create 2014-08-22 19:27:32,611.611 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:27:32,631.631 INFO:tasks.rados.rados.0.mira097.stdout:1062: finishing write tid 1 to mira09725123-389 2014-08-22 19:27:32,640.640 INFO:tasks.rados.rados.0.mira097.stdout:1062: finishing write tid 2 to mira09725123-389 2014-08-22 19:27:32,641.641 INFO:tasks.rados.rados.0.mira097.stdout:1062: finishing write tid 3 to mira09725123-389 2014-08-22 19:27:32,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1062: finishing write tid 5 to mira09725123-389 2014-08-22 19:27:32,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1062: finishing write tid 6 to mira09725123-389 2014-08-22 19:27:32,725.725 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 389 v 1689 (ObjNum 709 snap 52 seq_num 709) dirty exists 2014-08-22 19:27:32,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1062: done (1 left) 2014-08-22 19:27:32,726.726 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:32,826.826 INFO:tasks.rados.rados.0.mira097.stdout:1063: expect (ObjNum 87 snap 0 seq_num 87) 2014-08-22 19:27:33,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1063: done (0 left) 2014-08-22 19:27:33,570.570 INFO:tasks.rados.rados.0.mira097.stdout:1064: done (0 left) 2014-08-22 19:27:33,570.570 INFO:tasks.rados.rados.0.mira097.stdout:1065: write oid 490 current snap is 53 2014-08-22 19:27:33,570.570 INFO:tasks.rados.rados.0.mira097.stdout:1065: seq_num 710 ranges {538429=621371,1591000=612772,2782863=555566} 2014-08-22 19:27:33,595.595 INFO:tasks.rados.rados.0.mira097.stdout:1065: writing mira09725123-490 from 538429 to 1159800 tid 1 2014-08-22 19:27:33,617.617 INFO:tasks.rados.rados.0.mira097.stdout:1065: writing mira09725123-490 from 1591000 to 2203772 tid 2 2014-08-22 19:27:33,638.638 INFO:tasks.rados.rados.0.mira097.stdout:1065: writing mira09725123-490 from 2782863 to 3338429 tid 3 2014-08-22 19:27:33,640.640 INFO:tasks.rados.rados.0.mira097.stdout:1066: copy_from oid 433 from oid 153 current snap is 53 2014-08-22 19:27:33,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1065: finishing write tid 1 to mira09725123-490 2014-08-22 19:27:33,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1065: finishing write tid 2 to mira09725123-490 2014-08-22 19:27:33,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1067: copy_from oid 189 from oid 192 current snap is 53 2014-08-22 19:27:33,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1068: rollback oid 430 current snap is 53 2014-08-22 19:27:33,643.643 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 430 to 52 2014-08-22 19:27:33,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1069: read oid 42 snap 52 2014-08-22 19:27:33,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1070: delete oid 137 current snap is 53 2014-08-22 19:27:33,647.647 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 430 v 1640 (ObjNum 490 snap 0 seq_num 490) dirty exists 2014-08-22 19:27:33,669.669 INFO:tasks.rados.rados.0.mira097.stdout:1065: finishing write tid 3 to mira09725123-490 2014-08-22 19:27:33,671.671 INFO:tasks.rados.rados.0.mira097.stdout:1065: finishing write tid 5 to mira09725123-490 2014-08-22 19:27:33,672.672 INFO:tasks.rados.rados.0.mira097.stdout:1065: finishing write tid 6 to mira09725123-490 2014-08-22 19:27:33,672.672 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 490 v 1694 (ObjNum 710 snap 53 seq_num 710) dirty exists 2014-08-22 19:27:33,701.701 INFO:tasks.rados.rados.0.mira097.stdout:1065: done (5 left) 2014-08-22 19:27:33,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1068: done (4 left) 2014-08-22 19:27:33,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1070: done (3 left) 2014-08-22 19:27:33,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1071: write oid 109 current snap is 53 2014-08-22 19:27:33,702.702 INFO:tasks.rados.rados.0.mira097.stdout:1071: seq_num 712 ranges {577464=717427,1783167=787210,3184288=593176} 2014-08-22 19:27:33,741.741 INFO:tasks.rados.rados.0.mira097.stdout:1071: writing mira09725123-109 from 577464 to 1294891 tid 1 2014-08-22 19:27:33,760.760 INFO:tasks.rados.rados.0.mira097.stdout:1071: writing mira09725123-109 from 1783167 to 2570377 tid 2 2014-08-22 19:27:33,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1071: writing mira09725123-109 from 3184288 to 3777464 tid 3 2014-08-22 19:27:33,786.786 INFO:tasks.rados.rados.0.mira097.stdout:1072: snap_remove snap 50 2014-08-22 19:27:33,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1066: finishing copy_from to mira09725123-433 2014-08-22 19:27:33,787.787 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 433 v 1357 (ObjNum 152 snap 0 seq_num 152) dirty exists 2014-08-22 19:27:33,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1066: finishing copy_from racing read to mira09725123-433 2014-08-22 19:27:33,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1069: expect (ObjNum 8 snap 0 seq_num 8) 2014-08-22 19:27:35,046.046 INFO:tasks.rados.rados.0.mira097.stdout:1067: finishing copy_from to mira09725123-189 2014-08-22 19:27:35,046.046 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 189 v 248 (ObjNum 191 snap 0 seq_num 191) dirty exists 2014-08-22 19:27:35,046.046 INFO:tasks.rados.rados.0.mira097.stdout:1071: finishing write tid 1 to mira09725123-109 2014-08-22 19:27:35,046.046 INFO:tasks.rados.rados.0.mira097.stdout:1071: finishing write tid 2 to mira09725123-109 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1071: finishing write tid 3 to mira09725123-109 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1071: finishing write tid 5 to mira09725123-109 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1067: finishing copy_from racing read to mira09725123-189 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1071: finishing write tid 6 to mira09725123-109 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 109 v 1599 (ObjNum 712 snap 53 seq_num 712) dirty exists 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1066: done (4 left) 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1067: done (3 left) 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1069: done (2 left) 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1071: done (1 left) 2014-08-22 19:27:35,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1072: done (0 left) 2014-08-22 19:27:35,048.048 INFO:tasks.rados.rados.0.mira097.stdout:1073: delete oid 48 current snap is 53 2014-08-22 19:27:35,054.054 INFO:tasks.rados.rados.0.mira097.stdout:1073: done (0 left) 2014-08-22 19:27:35,054.054 INFO:tasks.rados.rados.0.mira097.stdout:1074: write oid 273 current snap is 53 2014-08-22 19:27:35,054.054 INFO:tasks.rados.rados.0.mira097.stdout:1074: seq_num 714 ranges {733875=1} 2014-08-22 19:27:35,063.063 INFO:tasks.rados.rados.0.mira097.stdout:1074: writing mira09725123-273 from 733875 to 733876 tid 1 2014-08-22 19:27:35,063.063 INFO:tasks.rados.rados.0.mira097.stdout:1075: snap_create 2014-08-22 19:27:35,063.063 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:35,243.243 INFO:tasks.rados.rados.0.mira097.stdout:1074: finishing write tid 1 to mira09725123-273 2014-08-22 19:27:35,244.244 INFO:tasks.rados.rados.0.mira097.stdout:1074: finishing write tid 3 to mira09725123-273 2014-08-22 19:27:35,245.245 INFO:tasks.rados.rados.0.mira097.stdout:1074: finishing write tid 4 to mira09725123-273 2014-08-22 19:27:35,246.246 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 273 v 1479 (ObjNum 714 snap 53 seq_num 714) dirty exists 2014-08-22 19:27:35,246.246 INFO:tasks.rados.rados.0.mira097.stdout:1074: done (0 left) 2014-08-22 19:27:35,586.586 INFO:tasks.rados.rados.0.mira097.stdout:1075: done (0 left) 2014-08-22 19:27:35,586.586 INFO:tasks.rados.rados.0.mira097.stdout:1076: snap_remove snap 53 2014-08-22 19:27:36,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1076: done (0 left) 2014-08-22 19:27:36,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1077: write oid 321 current snap is 54 2014-08-22 19:27:36,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1077: seq_num 715 ranges {678706=668936,1478705=1} 2014-08-22 19:27:36,628.628 INFO:tasks.rados.rados.0.mira097.stdout:1077: writing mira09725123-321 from 678706 to 1347642 tid 1 2014-08-22 19:27:36,629.629 INFO:tasks.rados.rados.0.mira097.stdout:1077: writing mira09725123-321 from 1478705 to 1478706 tid 2 2014-08-22 19:27:36,629.629 INFO:tasks.rados.rados.0.mira097.stdout:1078: read oid 168 snap -1 2014-08-22 19:27:36,629.629 INFO:tasks.rados.rados.0.mira097.stdout:1079: delete oid 346 current snap is 54 2014-08-22 19:27:36,641.641 INFO:tasks.rados.rados.0.mira097.stdout:1079: done (2 left) 2014-08-22 19:27:36,641.641 INFO:tasks.rados.rados.0.mira097.stdout:1080: read oid 489 snap -1 2014-08-22 19:27:36,641.641 INFO:tasks.rados.rados.0.mira097.stdout:1081: snap_create 2014-08-22 19:27:36,642.642 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:27:36,665.665 INFO:tasks.rados.rados.0.mira097.stdout:1077: finishing write tid 1 to mira09725123-321 2014-08-22 19:27:36,667.667 INFO:tasks.rados.rados.0.mira097.stdout:1077: finishing write tid 2 to mira09725123-321 2014-08-22 19:27:36,668.668 INFO:tasks.rados.rados.0.mira097.stdout:1077: finishing write tid 4 to mira09725123-321 2014-08-22 19:27:36,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1077: finishing write tid 5 to mira09725123-321 2014-08-22 19:27:36,978.978 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 321 v 1488 (ObjNum 715 snap 54 seq_num 715) dirty exists 2014-08-22 19:27:36,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1077: done (2 left) 2014-08-22 19:27:36,978.978 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:27:37,016.016 INFO:tasks.rados.rados.0.mira097.stdout:1080: expect (ObjNum 488 snap 0 seq_num 488) 2014-08-22 19:27:37,703.703 INFO:tasks.rados.rados.0.mira097.stdout:1078: expect (ObjNum 167 snap 0 seq_num 167) 2014-08-22 19:27:38,070.070 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:38,071.071 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:38,071.071 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.123839 2014-08-22 19:27:38,071.071 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.123838746144' 2014-08-22 19:27:38,543.543 INFO:tasks.rados.rados.0.mira097.stdout:1078: done (1 left) 2014-08-22 19:27:38,543.543 INFO:tasks.rados.rados.0.mira097.stdout:1080: done (0 left) 2014-08-22 19:27:38,705.705 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.583752 7f6cd612f700 1 librados: starting msgr at :/0 2014-08-22 19:27:38,705.705 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.583759 7f6cd612f700 1 librados: starting objecter 2014-08-22 19:27:38,705.705 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.583916 7f6cd612f700 1 librados: setting wanted keys 2014-08-22 19:27:38,705.705 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.583920 7f6cd612f700 1 librados: calling monclient init 2014-08-22 19:27:38,706.706 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.584706 7f6cceffd700 10 client.?.objecter ms_handle_connect 0x7f6cd00355e0 2014-08-22 19:27:38,706.706 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.584731 7f6cceffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:38,706.706 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.587178 7f6cd612f700 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:38,706.706 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.587383 7f6cd612f700 1 librados: init done 2014-08-22 19:27:38,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589350 7f6cceffd700 10 client.4221.objecter ms_dispatch 0x7f6cd0000f80 osd_map(162..162 src has 1..162) v3 2014-08-22 19:27:38,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589356 7f6cceffd700 3 client.4221.objecter handle_osd_map got epochs [162,162] > 0 2014-08-22 19:27:38,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589358 7f6cceffd700 3 client.4221.objecter handle_osd_map decoding full epoch 162 2014-08-22 19:27:38,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589425 7f6cceffd700 20 client.4221.objecter dump_active .. 0 homeless 2014-08-22 19:27:38,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589448 7f6cceffd700 10 client.4221.objecter ms_dispatch 0x7f6cd0000f80 osd_map(162..162 src has 1..162) v3 2014-08-22 19:27:38,708.708 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589450 7f6cceffd700 3 client.4221.objecter handle_osd_map ignoring epochs [162,162] <= 162 2014-08-22 19:27:38,708.708 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:31.589452 7f6cceffd700 20 client.4221.objecter dump_active .. 0 homeless 2014-08-22 19:27:38,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1081: done (0 left) 2014-08-22 19:27:38,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1082: read oid 434 snap -1 2014-08-22 19:27:38,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1083: snap_remove snap 43 2014-08-22 19:27:38,931.931 INFO:tasks.rados.rados.0.mira097.stdout:1082: expect (ObjNum 433 snap 0 seq_num 433) 2014-08-22 19:27:39,743.743 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.123839 (881152) 2014-08-22 19:27:39,747.747 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:32.649538 7f6cd612f700 20 client.4221.objecter shutdown clearing up homeless session... 2014-08-22 19:27:39,747.747 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:32.650130 7f6cd612f700 1 librados: shutdown 2014-08-22 19:27:40,147.147 INFO:tasks.rados.rados.0.mira097.stdout:1082: done (1 left) 2014-08-22 19:27:40,147.147 INFO:tasks.rados.rados.0.mira097.stdout:1083: done (0 left) 2014-08-22 19:27:40,147.147 INFO:tasks.rados.rados.0.mira097.stdout:1084: snap_remove snap 54 2014-08-22 19:27:40,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1084: done (0 left) 2014-08-22 19:27:40,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1085: write oid 190 current snap is 55 2014-08-22 19:27:40,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1085: seq_num 717 ranges {435117=799041,2033060=402057} 2014-08-22 19:27:40,742.742 INFO:tasks.rados.rados.0.mira097.stdout:1085: writing mira09725123-190 from 435117 to 1234158 tid 1 2014-08-22 19:27:40,759.759 INFO:tasks.rados.rados.0.mira097.stdout:1085: writing mira09725123-190 from 2033060 to 2435117 tid 2 2014-08-22 19:27:40,761.761 INFO:tasks.rados.rados.0.mira097.stdout:1086: snap_create 2014-08-22 19:27:40,761.761 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:40,776.776 INFO:tasks.rados.rados.0.mira097.stdout:1085: finishing write tid 1 to mira09725123-190 2014-08-22 19:27:40,780.780 INFO:tasks.rados.rados.0.mira097.stdout:1085: finishing write tid 2 to mira09725123-190 2014-08-22 19:27:40,781.781 INFO:tasks.rados.rados.0.mira097.stdout:1085: finishing write tid 4 to mira09725123-190 2014-08-22 19:27:40,782.782 INFO:tasks.rados.rados.0.mira097.stdout:1085: finishing write tid 5 to mira09725123-190 2014-08-22 19:27:40,782.782 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 190 v 1745 (ObjNum 717 snap 55 seq_num 717) dirty exists 2014-08-22 19:27:40,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1085: done (0 left) 2014-08-22 19:27:41,717.717 INFO:tasks.rados.rados.0.mira097.stdout:1086: done (0 left) 2014-08-22 19:27:41,717.717 INFO:tasks.rados.rados.0.mira097.stdout:1087: copy_from oid 341 from oid 294 current snap is 56 2014-08-22 19:27:41,717.717 INFO:tasks.rados.rados.0.mira097.stdout:1088: delete oid 290 current snap is 56 2014-08-22 19:27:41,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1088: done (1 left) 2014-08-22 19:27:41,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1089: read oid 49 snap -1 2014-08-22 19:27:41,736.736 INFO:tasks.rados.rados.0.mira097.stdout:1090: write oid 263 current snap is 56 2014-08-22 19:27:41,736.736 INFO:tasks.rados.rados.0.mira097.stdout:1090: seq_num 719 ranges {722729=743674,1922381=532849,3090020=615821,3922728=1} 2014-08-22 19:27:41,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1090: writing mira09725123-263 from 722729 to 1466403 tid 1 2014-08-22 19:27:41,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1090: writing mira09725123-263 from 1922381 to 2455230 tid 2 2014-08-22 19:27:41,807.807 INFO:tasks.rados.rados.0.mira097.stdout:1090: writing mira09725123-263 from 3090020 to 3705841 tid 3 2014-08-22 19:27:41,808.808 INFO:tasks.rados.rados.0.mira097.stdout:1090: writing mira09725123-263 from 3922728 to 3922729 tid 4 2014-08-22 19:27:41,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1091: read oid 116 snap -1 2014-08-22 19:27:41,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1092: snap_create 2014-08-22 19:27:41,889.889 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:27:41,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1091: expect (ObjNum 115 snap 0 seq_num 115) 2014-08-22 19:27:41,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1089: expect (ObjNum 48 snap 0 seq_num 48) 2014-08-22 19:27:42,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 1 to mira09725123-263 2014-08-22 19:27:42,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1087: finishing copy_from to mira09725123-341 2014-08-22 19:27:42,743.743 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 341 v 385 (ObjNum 293 snap 0 seq_num 293) dirty exists 2014-08-22 19:27:42,743.743 INFO:tasks.rados.rados.0.mira097.stdout:1087: finishing copy_from racing read to mira09725123-341 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 2 to mira09725123-263 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 3 to mira09725123-263 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 4 to mira09725123-263 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 6 to mira09725123-263 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: finishing write tid 7 to mira09725123-263 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 263 v 1756 (ObjNum 719 snap 56 seq_num 719) dirty exists 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1087: done (3 left) 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1089: done (2 left) 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1090: done (1 left) 2014-08-22 19:27:42,744.744 INFO:tasks.rados.rados.0.mira097.stdout:1091: done (0 left) 2014-08-22 19:27:43,738.738 INFO:tasks.rados.rados.0.mira097.stdout:1092: done (0 left) 2014-08-22 19:27:43,738.738 INFO:tasks.rados.rados.0.mira097.stdout:1093: write oid 380 current snap is 57 2014-08-22 19:27:43,738.738 INFO:tasks.rados.rados.0.mira097.stdout:1093: seq_num 720 ranges {418982=715478,1636513=451620,2739821=501587,3618981=1} 2014-08-22 19:27:43,767.767 INFO:tasks.rados.rados.0.mira097.stdout:1093: writing mira09725123-380 from 418982 to 1134460 tid 1 2014-08-22 19:27:43,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1093: writing mira09725123-380 from 1636513 to 2088133 tid 2 2014-08-22 19:27:43,804.804 INFO:tasks.rados.rados.0.mira097.stdout:1093: writing mira09725123-380 from 2739821 to 3241408 tid 3 2014-08-22 19:27:43,807.807 INFO:tasks.rados.rados.0.mira097.stdout:1093: writing mira09725123-380 from 3618981 to 3618982 tid 4 2014-08-22 19:27:43,807.807 INFO:tasks.rados.rados.0.mira097.stdout:1094: snap_create 2014-08-22 19:27:43,808.808 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:43,829.829 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 1 to mira09725123-380 2014-08-22 19:27:43,830.830 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 2 to mira09725123-380 2014-08-22 19:27:43,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 3 to mira09725123-380 2014-08-22 19:27:43,913.913 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 4 to mira09725123-380 2014-08-22 19:27:43,913.913 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 6 to mira09725123-380 2014-08-22 19:27:43,913.913 INFO:tasks.rados.rados.0.mira097.stdout:1093: finishing write tid 7 to mira09725123-380 2014-08-22 19:27:43,914.914 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 380 v 1766 (ObjNum 720 snap 57 seq_num 720) dirty exists 2014-08-22 19:27:43,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1093: done (0 left) 2014-08-22 19:27:44,748.748 INFO:tasks.rados.rados.0.mira097.stdout:1094: done (0 left) 2014-08-22 19:27:44,749.749 INFO:tasks.rados.rados.0.mira097.stdout:1095: snap_remove snap 57 2014-08-22 19:27:44,753.753 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:44,753.753 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:44,754.754 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:27:44,754.754 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:27:45,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.134005 7febde373700 1 librados: starting msgr at :/0 2014-08-22 19:27:45,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.134017 7febde373700 1 librados: starting objecter 2014-08-22 19:27:45,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.134249 7febde373700 1 librados: setting wanted keys 2014-08-22 19:27:45,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.134255 7febde373700 1 librados: calling monclient init 2014-08-22 19:27:45,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.136579 7febd6ffd700 10 client.?.objecter ms_handle_connect 0x7febd80355e0 2014-08-22 19:27:45,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.136617 7febd6ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:45,235.235 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.140467 7febde373700 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:45,236.236 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.140613 7febde373700 1 librados: init done 2014-08-22 19:27:45,236.236 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141335 7febd6ffd700 10 client.4219.objecter ms_dispatch 0x7febd8000f80 osd_map(168..168 src has 1..168) v3 2014-08-22 19:27:45,236.236 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141355 7febd6ffd700 3 client.4219.objecter handle_osd_map got epochs [168,168] > 0 2014-08-22 19:27:45,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141360 7febd6ffd700 3 client.4219.objecter handle_osd_map decoding full epoch 168 2014-08-22 19:27:45,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141550 7febd6ffd700 20 client.4219.objecter dump_active .. 0 homeless 2014-08-22 19:27:45,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141640 7febd6ffd700 10 client.4219.objecter ms_dispatch 0x7febd8000f80 osd_map(168..168 src has 1..168) v3 2014-08-22 19:27:45,237.237 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141654 7febd6ffd700 3 client.4219.objecter handle_osd_map ignoring epochs [168,168] <= 168 2014-08-22 19:27:45,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.141659 7febd6ffd700 20 client.4219.objecter dump_active .. 0 homeless 2014-08-22 19:27:45,413.413 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:27:45,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.251501 7febde373700 20 client.4219.objecter shutdown clearing up homeless session... 2014-08-22 19:27:45,690.690 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:38.252595 7febde373700 1 librados: shutdown 2014-08-22 19:27:45,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1095: done (0 left) 2014-08-22 19:27:45,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1096: delete oid 24 current snap is 58 2014-08-22 19:27:45,865.865 INFO:tasks.rados.rados.0.mira097.stdout:1096: done (0 left) 2014-08-22 19:27:45,865.865 INFO:tasks.rados.rados.0.mira097.stdout:1097: copy_from oid 458 from oid 147 current snap is 58 2014-08-22 19:27:45,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1098: read oid 438 snap -1 2014-08-22 19:27:45,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1099: read oid 296 snap -1 2014-08-22 19:27:45,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1100: snap_remove snap 56 2014-08-22 19:27:45,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1098: expect (ObjNum 437 snap 0 seq_num 437) 2014-08-22 19:27:45,874.874 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 74' 2014-08-22 19:27:46,357.357 INFO:tasks.rados.rados.0.mira097.stdout:1099: expect (ObjNum 295 snap 0 seq_num 295) 2014-08-22 19:27:46,360.360 INFO:tasks.rados.rados.0.mira097.stdout:1097: finishing copy_from to mira09725123-458 2014-08-22 19:27:46,361.361 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 458 v 939 (ObjNum 577 snap 23 seq_num 577) dirty exists 2014-08-22 19:27:46,361.361 INFO:tasks.rados.rados.0.mira097.stdout:1097: finishing copy_from racing read to mira09725123-458 2014-08-22 19:27:46,553.553 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.457917 7f242e3ad700 1 librados: starting msgr at :/0 2014-08-22 19:27:46,553.553 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.457928 7f242e3ad700 1 librados: starting objecter 2014-08-22 19:27:46,554.554 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.458184 7f242e3ad700 1 librados: setting wanted keys 2014-08-22 19:27:46,554.554 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.458199 7f242e3ad700 1 librados: calling monclient init 2014-08-22 19:27:46,554.554 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.459673 7f2426ffd700 10 client.?.objecter ms_handle_connect 0x7f24280355e0 2014-08-22 19:27:46,555.555 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.459711 7f2426ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:46,559.559 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.463131 7f242e3ad700 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:46,559.559 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.463333 7f242e3ad700 1 librados: init done 2014-08-22 19:27:46,559.559 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.463970 7f2426ffd700 10 client.4208.objecter ms_dispatch 0x7f2428000f80 osd_map(169..169 src has 1..169) v3 2014-08-22 19:27:46,559.559 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.463983 7f2426ffd700 3 client.4208.objecter handle_osd_map got epochs [169,169] > 0 2014-08-22 19:27:46,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.463987 7f2426ffd700 3 client.4208.objecter handle_osd_map decoding full epoch 169 2014-08-22 19:27:46,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.464162 7f2426ffd700 20 client.4208.objecter dump_active .. 0 homeless 2014-08-22 19:27:46,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.464262 7f2426ffd700 10 client.4208.objecter ms_dispatch 0x7f2428000f80 osd_map(169..169 src has 1..169) v3 2014-08-22 19:27:46,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.464276 7f2426ffd700 3 client.4208.objecter handle_osd_map ignoring epochs [169,169] <= 169 2014-08-22 19:27:46,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.464282 7f2426ffd700 20 client.4208.objecter dump_active .. 0 homeless 2014-08-22 19:27:46,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1097: done (3 left) 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1098: done (2 left) 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1099: done (1 left) 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1100: done (0 left) 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1101: read oid 354 snap -1 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1102: read oid 77 snap -1 2014-08-22 19:27:46,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1103: write oid 325 current snap is 58 2014-08-22 19:27:46,765.765 INFO:tasks.rados.rados.0.mira097.stdout:1103: seq_num 722 ranges {423970=695183,1623969=1} 2014-08-22 19:27:46,797.797 INFO:tasks.rados.rados.0.mira097.stdout:1103: writing mira09725123-325 from 423970 to 1119153 tid 1 2014-08-22 19:27:46,802.802 INFO:tasks.rados.rados.0.mira097.stdout:1103: writing mira09725123-325 from 1623969 to 1623970 tid 2 2014-08-22 19:27:46,802.802 INFO:tasks.rados.rados.0.mira097.stdout:1104: write oid 197 current snap is 58 2014-08-22 19:27:46,802.802 INFO:tasks.rados.rados.0.mira097.stdout:1104: seq_num 723 ranges {520224=418411,1320223=1} 2014-08-22 19:27:46,808.808 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 74 2014-08-22 19:27:46,812.812 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.716936 7f242e3ad700 20 client.4208.objecter shutdown clearing up homeless session... 2014-08-22 19:27:46,812.812 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:39.717345 7f242e3ad700 1 librados: shutdown 2014-08-22 19:27:46,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1104: writing mira09725123-197 from 520224 to 938635 tid 1 2014-08-22 19:27:46,821.821 INFO:tasks.rados.rados.0.mira097.stdout:1104: writing mira09725123-197 from 1320223 to 1320224 tid 2 2014-08-22 19:27:46,822.822 INFO:tasks.rados.rados.0.mira097.stdout:1105: write oid 460 current snap is 58 2014-08-22 19:27:46,822.822 INFO:tasks.rados.rados.0.mira097.stdout:1101: expect (ObjNum 676 snap 51 seq_num 676) 2014-08-22 19:27:47,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1103: finishing write tid 1 to mira09725123-325 2014-08-22 19:27:47,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1103: finishing write tid 2 to mira09725123-325 2014-08-22 19:27:47,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1103: finishing write tid 4 to mira09725123-325 2014-08-22 19:27:47,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1103: finishing write tid 5 to mira09725123-325 2014-08-22 19:27:47,538.538 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 325 v 1772 (ObjNum 722 snap 58 seq_num 722) dirty exists 2014-08-22 19:27:47,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1104: finishing write tid 1 to mira09725123-197 2014-08-22 19:27:47,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1104: finishing write tid 2 to mira09725123-197 2014-08-22 19:27:47,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1104: finishing write tid 4 to mira09725123-197 2014-08-22 19:27:47,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1105: seq_num 724 ranges {464029=790215,1869425=194604} 2014-08-22 19:27:47,564.564 INFO:tasks.rados.rados.0.mira097.stdout:1105: writing mira09725123-460 from 464029 to 1254244 tid 1 2014-08-22 19:27:47,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1105: writing mira09725123-460 from 1869425 to 2064029 tid 2 2014-08-22 19:27:47,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1101: done (4 left) 2014-08-22 19:27:47,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1102: done (3 left) 2014-08-22 19:27:47,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1103: done (2 left) 2014-08-22 19:27:47,575.575 INFO:tasks.rados.rados.0.mira097.stdout:1106: copy_from oid 380 from oid 214 current snap is 58 2014-08-22 19:27:47,576.576 INFO:tasks.rados.rados.0.mira097.stdout:1104: finishing write tid 5 to mira09725123-197 2014-08-22 19:27:47,576.576 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 197 v 1627 (ObjNum 723 snap 58 seq_num 723) dirty exists 2014-08-22 19:27:47,576.576 INFO:tasks.rados.rados.0.mira097.stdout:1104: done (2 left) 2014-08-22 19:27:47,576.576 INFO:tasks.rados.rados.0.mira097.stdout:1107: snap_create 2014-08-22 19:27:47,576.576 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:27:47,604.604 INFO:tasks.rados.rados.0.mira097.stdout:1106: finishing copy_from to mira09725123-380 2014-08-22 19:27:47,604.604 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 380 v 295 (ObjNum 213 snap 0 seq_num 213) dirty exists 2014-08-22 19:27:47,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1106: finishing copy_from racing read to mira09725123-380 2014-08-22 19:27:47,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1106: done (1 left) 2014-08-22 19:27:47,606.606 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:47,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1105: finishing write tid 1 to mira09725123-460 2014-08-22 19:27:47,610.610 INFO:tasks.rados.rados.0.mira097.stdout:1105: finishing write tid 2 to mira09725123-460 2014-08-22 19:27:47,615.615 INFO:tasks.rados.rados.0.mira097.stdout:1105: finishing write tid 4 to mira09725123-460 2014-08-22 19:27:47,619.619 INFO:tasks.rados.rados.0.mira097.stdout:1105: finishing write tid 5 to mira09725123-460 2014-08-22 19:27:47,619.619 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 460 v 1519 (ObjNum 724 snap 58 seq_num 724) dirty exists 2014-08-22 19:27:47,619.619 INFO:tasks.rados.rados.0.mira097.stdout:1105: done (0 left) 2014-08-22 19:27:47,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1107: done (0 left) 2014-08-22 19:27:47,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1108: snap_remove snap 55 2014-08-22 19:27:48,803.803 INFO:tasks.rados.rados.0.mira097.stdout:1108: done (0 left) 2014-08-22 19:27:48,803.803 INFO:tasks.rados.rados.0.mira097.stdout:1109: write oid 16 current snap is 59 2014-08-22 19:27:48,803.803 INFO:tasks.rados.rados.0.mira097.stdout:1109: seq_num 725 ranges {408859=798116,1868230=589092,2808858=1} 2014-08-22 19:27:48,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1109: writing mira09725123-16 from 408859 to 1206975 tid 1 2014-08-22 19:27:48,861.861 INFO:tasks.rados.rados.0.mira097.stdout:1109: writing mira09725123-16 from 1868230 to 2457322 tid 2 2014-08-22 19:27:48,865.865 INFO:tasks.rados.rados.0.mira097.stdout:1109: writing mira09725123-16 from 2808858 to 2808859 tid 3 2014-08-22 19:27:48,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1110: write oid 18 current snap is 59 2014-08-22 19:27:48,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1110: seq_num 726 ranges {620441=769920,2097690=410711,3020440=1} 2014-08-22 19:27:48,891.891 INFO:tasks.rados.rados.0.mira097.stdout:1110: writing mira09725123-18 from 620441 to 1390361 tid 1 2014-08-22 19:27:48,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1110: writing mira09725123-18 from 2097690 to 2508401 tid 2 2014-08-22 19:27:48,913.913 INFO:tasks.rados.rados.0.mira097.stdout:1110: writing mira09725123-18 from 3020440 to 3020441 tid 3 2014-08-22 19:27:48,913.913 INFO:tasks.rados.rados.0.mira097.stdout:1111: copy_from oid 430 from oid 344 current snap is 59 2014-08-22 19:27:48,977.977 INFO:tasks.rados.rados.0.mira097.stdout:1109: finishing write tid 1 to mira09725123-16 2014-08-22 19:27:48,977.977 INFO:tasks.rados.rados.0.mira097.stdout:1109: finishing write tid 2 to mira09725123-16 2014-08-22 19:27:48,977.977 INFO:tasks.rados.rados.0.mira097.stdout:1109: finishing write tid 3 to mira09725123-16 2014-08-22 19:27:48,977.977 INFO:tasks.rados.rados.0.mira097.stdout:1109: finishing write tid 5 to mira09725123-16 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1109: finishing write tid 6 to mira09725123-16 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 16 v 1635 (ObjNum 725 snap 59 seq_num 725) dirty exists 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1109: done (2 left) 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1112: write oid 66 current snap is 59 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1112: seq_num 727 ranges {564247=493148,1775726=480906,3045550=546881,3764246=1} 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1112: writing mira09725123-66 from 564247 to 1057395 tid 1 2014-08-22 19:27:48,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1112: writing mira09725123-66 from 1775726 to 2256632 tid 2 2014-08-22 19:27:48,979.979 INFO:tasks.rados.rados.0.mira097.stdout:1112: writing mira09725123-66 from 3045550 to 3592431 tid 3 2014-08-22 19:27:48,981.981 INFO:tasks.rados.rados.0.mira097.stdout:1112: writing mira09725123-66 from 3764246 to 3764247 tid 4 2014-08-22 19:27:48,983.983 INFO:tasks.rados.rados.0.mira097.stdout:1113: delete oid 367 current snap is 59 2014-08-22 19:27:48,983.983 INFO:tasks.rados.rados.0.mira097.stdout:1111: finishing copy_from to mira09725123-430 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1111: got expected ENOENT (src dne) 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1111: finishing copy_from racing read to mira09725123-430 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1110: finishing write tid 1 to mira09725123-18 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1110: finishing write tid 2 to mira09725123-18 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1110: finishing write tid 3 to mira09725123-18 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1110: finishing write tid 5 to mira09725123-18 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1110: finishing write tid 6 to mira09725123-18 2014-08-22 19:27:48,984.984 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 18 v 1544 (ObjNum 726 snap 59 seq_num 726) dirty exists 2014-08-22 19:27:48,985.985 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 1 to mira09725123-66 2014-08-22 19:27:48,985.985 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 2 to mira09725123-66 2014-08-22 19:27:49,013.013 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 3 to mira09725123-66 2014-08-22 19:27:49,047.047 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 4 to mira09725123-66 2014-08-22 19:27:49,087.087 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 6 to mira09725123-66 2014-08-22 19:27:49,088.088 INFO:tasks.rados.rados.0.mira097.stdout:1112: finishing write tid 7 to mira09725123-66 2014-08-22 19:27:49,088.088 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 66 v 1793 (ObjNum 727 snap 59 seq_num 727) dirty exists 2014-08-22 19:27:49,096.096 INFO:tasks.rados.rados.0.mira097.stdout:1110: done (3 left) 2014-08-22 19:27:49,096.096 INFO:tasks.rados.rados.0.mira097.stdout:1111: done (2 left) 2014-08-22 19:27:49,096.096 INFO:tasks.rados.rados.0.mira097.stdout:1112: done (1 left) 2014-08-22 19:27:49,096.096 INFO:tasks.rados.rados.0.mira097.stdout:1113: done (0 left) 2014-08-22 19:27:49,097.097 INFO:tasks.rados.rados.0.mira097.stdout:1114: rollback oid 151 current snap is 59 2014-08-22 19:27:49,097.097 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 151 to 52 2014-08-22 19:27:49,097.097 INFO:tasks.rados.rados.0.mira097.stdout:1115: write oid 412 current snap is 59 2014-08-22 19:27:49,097.097 INFO:tasks.rados.rados.0.mira097.stdout:1115: seq_num 729 ranges {204305=1} 2014-08-22 19:27:49,098.098 INFO:tasks.rados.rados.0.mira097.stdout:1115: writing mira09725123-412 from 204305 to 204306 tid 1 2014-08-22 19:27:49,098.098 INFO:tasks.rados.rados.0.mira097.stdout:1116: snap_create 2014-08-22 19:27:49,098.098 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:27:49,139.139 INFO:tasks.rados.rados.0.mira097.stdout:1115: finishing write tid 1 to mira09725123-412 2014-08-22 19:27:49,140.140 INFO:tasks.rados.rados.0.mira097.stdout:1115: finishing write tid 3 to mira09725123-412 2014-08-22 19:27:49,140.140 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 151 v 1804 (ObjNum 540 snap 10 seq_num 540) dirty exists 2014-08-22 19:27:49,140.140 INFO:tasks.rados.rados.0.mira097.stdout:1114: done (1 left) 2014-08-22 19:27:49,140.140 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:49,148.148 INFO:tasks.rados.rados.0.mira097.stdout:1115: finishing write tid 4 to mira09725123-412 2014-08-22 19:27:49,149.149 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 412 v 1802 (ObjNum 729 snap 59 seq_num 729) dirty exists 2014-08-22 19:27:49,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1115: done (0 left) 2014-08-22 19:27:49,835.835 INFO:tasks.rados.rados.0.mira097.stdout:1116: done (0 left) 2014-08-22 19:27:49,836.836 INFO:tasks.rados.rados.0.mira097.stdout:1117: copy_from oid 413 from oid 488 current snap is 60 2014-08-22 19:27:49,836.836 INFO:tasks.rados.rados.0.mira097.stdout:1118: snap_remove snap 59 2014-08-22 19:27:49,948.948 INFO:tasks.rados.rados.0.mira097.stdout:1117: finishing copy_from to mira09725123-413 2014-08-22 19:27:49,948.948 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 413 v 596 (ObjNum 487 snap 0 seq_num 487) dirty exists 2014-08-22 19:27:49,948.948 INFO:tasks.rados.rados.0.mira097.stdout:1117: finishing copy_from racing read to mira09725123-413 2014-08-22 19:27:50,849.849 INFO:tasks.rados.rados.0.mira097.stdout:1117: done (1 left) 2014-08-22 19:27:50,849.849 INFO:tasks.rados.rados.0.mira097.stdout:1118: done (0 left) 2014-08-22 19:27:50,849.849 INFO:tasks.rados.rados.0.mira097.stdout:1119: read oid 184 snap -1 2014-08-22 19:27:50,852.852 INFO:tasks.rados.rados.0.mira097.stdout:1120: rollback oid 178 current snap is 60 2014-08-22 19:27:50,853.853 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 178 to 52 2014-08-22 19:27:50,853.853 INFO:tasks.rados.rados.0.mira097.stdout:1121: write oid 409 current snap is 60 2014-08-22 19:27:50,853.853 INFO:tasks.rados.rados.0.mira097.stdout:1121: seq_num 730 ranges {549136=400000} 2014-08-22 19:27:50,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1121: writing mira09725123-409 from 549136 to 949136 tid 1 2014-08-22 19:27:50,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1122: write oid 72 current snap is 60 2014-08-22 19:27:50,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1122: seq_num 731 ranges {610318=416462,1741887=468431} 2014-08-22 19:27:50,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1122: writing mira09725123-72 from 610318 to 1026780 tid 1 2014-08-22 19:27:50,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1122: writing mira09725123-72 from 1741887 to 2210318 tid 2 2014-08-22 19:27:50,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1123: copy_from oid 74 from oid 214 current snap is 60 2014-08-22 19:27:50,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1121: finishing write tid 1 to mira09725123-409 2014-08-22 19:27:50,915.915 INFO:tasks.rados.rados.0.mira097.stdout:1121: finishing write tid 3 to mira09725123-409 2014-08-22 19:27:50,915.915 INFO:tasks.rados.rados.0.mira097.stdout:1121: finishing write tid 4 to mira09725123-409 2014-08-22 19:27:50,915.915 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 409 v 1549 (ObjNum 730 snap 60 seq_num 730) dirty exists 2014-08-22 19:27:50,915.915 INFO:tasks.rados.rados.0.mira097.stdout:1122: finishing write tid 1 to mira09725123-72 2014-08-22 19:27:51,024.024 INFO:tasks.rados.rados.0.mira097.stdout:1121: done (4 left) 2014-08-22 19:27:51,024.024 INFO:tasks.rados.rados.0.mira097.stdout:1124: read oid 418 snap -1 2014-08-22 19:27:51,024.024 INFO:tasks.rados.rados.0.mira097.stdout:1125: write oid 379 current snap is 60 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1125: seq_num 732 ranges {705547=789290,1905546=1} 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1125: writing mira09725123-379 from 705547 to 1494837 tid 1 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1125: writing mira09725123-379 from 1905546 to 1905547 tid 2 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1126: snap_create 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1122: finishing write tid 2 to mira09725123-72 2014-08-22 19:27:51,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1122: finishing write tid 4 to mira09725123-72 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1122: finishing write tid 5 to mira09725123-72 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 72 v 1552 (ObjNum 731 snap 60 seq_num 731) dirty exists 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1122: done (5 left) 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1123: finishing copy_from to mira09725123-74 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 74 v 295 (ObjNum 213 snap 0 seq_num 213) dirty exists 2014-08-22 19:27:51,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1123: finishing copy_from racing read to mira09725123-74 2014-08-22 19:27:51,027.027 INFO:tasks.rados.rados.0.mira097.stdout:1123: done (4 left) 2014-08-22 19:27:51,027.027 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:27:51,027.027 INFO:tasks.rados.rados.0.mira097.stdout:1119: expect (ObjNum 514 snap 5 seq_num 514) 2014-08-22 19:27:51,817.817 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:51,817.817 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:51,818.818 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:27:51,818.818 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:27:52,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.169908 7f579072b700 1 librados: starting msgr at :/0 2014-08-22 19:27:52,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.169915 7f579072b700 1 librados: starting objecter 2014-08-22 19:27:52,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.170054 7f579072b700 1 librados: setting wanted keys 2014-08-22 19:27:52,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.170060 7f579072b700 1 librados: calling monclient init 2014-08-22 19:27:52,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.170888 7f578d7d8700 10 client.?.objecter ms_handle_connect 0x7f57880355e0 2014-08-22 19:27:52,266.266 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.170909 7f578d7d8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:52,267.267 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172199 7f579072b700 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:52,267.267 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172299 7f579072b700 1 librados: init done 2014-08-22 19:27:52,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172583 7f578d7d8700 10 client.4211.objecter ms_dispatch 0x7f5788000f80 osd_map(174..174 src has 1..174) v3 2014-08-22 19:27:52,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172589 7f578d7d8700 3 client.4211.objecter handle_osd_map got epochs [174,174] > 0 2014-08-22 19:27:52,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172591 7f578d7d8700 3 client.4211.objecter handle_osd_map decoding full epoch 174 2014-08-22 19:27:52,369.369 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172658 7f578d7d8700 20 client.4211.objecter dump_active .. 0 homeless 2014-08-22 19:27:52,369.369 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172686 7f578d7d8700 10 client.4211.objecter ms_dispatch 0x7f5788000f80 osd_map(174..174 src has 1..174) v3 2014-08-22 19:27:52,370.370 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172688 7f578d7d8700 3 client.4211.objecter handle_osd_map ignoring epochs [174,174] <= 174 2014-08-22 19:27:52,370.370 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.172690 7f578d7d8700 20 client.4211.objecter dump_active .. 0 homeless 2014-08-22 19:27:52,449.449 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:27:52,895.895 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.261721 7f579072b700 20 client.4211.objecter shutdown clearing up homeless session... 2014-08-22 19:27:52,895.895 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:45.262412 7f579072b700 1 librados: shutdown 2014-08-22 19:27:52,895.895 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 178 v 1818 (ObjNum 177 snap 0 seq_num 177) dirty exists 2014-08-22 19:27:52,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1124: expect (ObjNum 40 snap 0 seq_num 40) 2014-08-22 19:27:53,083.083 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:27:53,084.084 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 84' 2014-08-22 19:27:53,736.736 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.640456 7f0afc48e700 1 librados: starting msgr at :/0 2014-08-22 19:27:53,736.736 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.640467 7f0afc48e700 1 librados: starting objecter 2014-08-22 19:27:53,736.736 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.640590 7f0afc48e700 1 librados: setting wanted keys 2014-08-22 19:27:53,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.640594 7f0afc48e700 1 librados: calling monclient init 2014-08-22 19:27:53,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.642377 7f0af953b700 10 client.?.objecter ms_handle_connect 0x7f0af40355e0 2014-08-22 19:27:53,737.737 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.642400 7f0af953b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:53,741.741 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.645721 7f0afc48e700 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:53,741.741 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.645811 7f0afc48e700 1 librados: init done 2014-08-22 19:27:53,742.742 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646602 7f0af953b700 10 client.4222.objecter ms_dispatch 0x7f0af4000f80 osd_map(174..174 src has 1..174) v3 2014-08-22 19:27:53,742.742 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646608 7f0af953b700 3 client.4222.objecter handle_osd_map got epochs [174,174] > 0 2014-08-22 19:27:53,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646610 7f0af953b700 3 client.4222.objecter handle_osd_map decoding full epoch 174 2014-08-22 19:27:53,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646684 7f0af953b700 20 client.4222.objecter dump_active .. 0 homeless 2014-08-22 19:27:53,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646719 7f0af953b700 10 client.4222.objecter ms_dispatch 0x7f0af4000f80 osd_map(174..174 src has 1..174) v3 2014-08-22 19:27:53,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646722 7f0af953b700 3 client.4222.objecter handle_osd_map ignoring epochs [174,174] <= 174 2014-08-22 19:27:53,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.646723 7f0af953b700 20 client.4222.objecter dump_active .. 0 homeless 2014-08-22 19:27:53,788.788 INFO:tasks.rados.rados.0.mira097.stdout:1125: finishing write tid 1 to mira09725123-379 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1125: finishing write tid 2 to mira09725123-379 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1125: finishing write tid 4 to mira09725123-379 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1125: finishing write tid 5 to mira09725123-379 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 379 v 1642 (ObjNum 732 snap 60 seq_num 732) dirty exists 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1119: done (3 left) 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1120: done (2 left) 2014-08-22 19:27:53,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1124: done (1 left) 2014-08-22 19:27:53,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1125: done (0 left) 2014-08-22 19:27:53,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1126: done (0 left) 2014-08-22 19:27:53,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1127: copy_from oid 332 from oid 92 current snap is 61 2014-08-22 19:27:53,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1128: read oid 110 snap -1 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:1129: read oid 468 snap 52 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:1130: read oid 362 snap -1 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:1131: rollback oid 176 current snap is 61 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 176 to 58 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:1132: write oid 304 current snap is 61 2014-08-22 19:27:53,874.874 INFO:tasks.rados.rados.0.mira097.stdout:1132: seq_num 733 ranges {650377=796167,2218727=431650} 2014-08-22 19:27:53,890.890 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 84 2014-08-22 19:27:53,893.893 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.794963 7f0afc48e700 20 client.4222.objecter shutdown clearing up homeless session... 2014-08-22 19:27:53,896.896 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:46.795407 7f0afc48e700 1 librados: shutdown 2014-08-22 19:27:53,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1132: writing mira09725123-304 from 650377 to 1446544 tid 1 2014-08-22 19:27:53,926.926 INFO:tasks.rados.rados.0.mira097.stdout:1132: writing mira09725123-304 from 2218727 to 2650377 tid 2 2014-08-22 19:27:53,927.927 INFO:tasks.rados.rados.0.mira097.stdout:1128: expect (ObjNum 109 snap 0 seq_num 109) 2014-08-22 19:27:54,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1129: expect (ObjNum 467 snap 0 seq_num 467) 2014-08-22 19:27:54,416.416 INFO:tasks.rados.rados.0.mira097.stdout:1132: finishing write tid 1 to mira09725123-304 2014-08-22 19:27:54,416.416 INFO:tasks.rados.rados.0.mira097.stdout:1130: expect (ObjNum 361 snap 0 seq_num 361) 2014-08-22 19:27:54,472.472 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 176 v 1650 (ObjNum 175 snap 0 seq_num 175) dirty exists 2014-08-22 19:27:54,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1132: finishing write tid 2 to mira09725123-304 2014-08-22 19:27:54,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1132: finishing write tid 4 to mira09725123-304 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1132: finishing write tid 5 to mira09725123-304 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 304 v 1675 (ObjNum 733 snap 61 seq_num 733) dirty exists 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1127: finishing copy_from to mira09725123-332 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 332 v 124 (ObjNum 91 snap 0 seq_num 91) dirty exists 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1127: finishing copy_from racing read to mira09725123-332 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1127: done (5 left) 2014-08-22 19:27:54,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1128: done (4 left) 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1129: done (3 left) 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1130: done (2 left) 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1131: done (1 left) 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1132: done (0 left) 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1133: copy_from oid 126 from oid 162 current snap is 61 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1134: rollback oid 81 current snap is 61 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 81 to 52 2014-08-22 19:27:54,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1135: read oid 423 snap -1 2014-08-22 19:27:54,475.475 INFO:tasks.rados.rados.0.mira097.stdout:1136: write oid 153 current snap is 61 2014-08-22 19:27:54,475.475 INFO:tasks.rados.rados.0.mira097.stdout:1136: seq_num 734 ranges {594183=768995,1897787=757744,3277808=116375} 2014-08-22 19:27:54,503.503 INFO:tasks.rados.rados.0.mira097.stdout:1136: writing mira09725123-153 from 594183 to 1363178 tid 1 2014-08-22 19:27:54,531.531 INFO:tasks.rados.rados.0.mira097.stdout:1136: writing mira09725123-153 from 1897787 to 2655531 tid 2 2014-08-22 19:27:54,540.540 INFO:tasks.rados.rados.0.mira097.stdout:1136: writing mira09725123-153 from 3277808 to 3394183 tid 3 2014-08-22 19:27:54,540.540 INFO:tasks.rados.rados.0.mira097.stdout:1137: rollback oid 431 current snap is 61 2014-08-22 19:27:54,540.540 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 431 to 58 2014-08-22 19:27:54,541.541 INFO:tasks.rados.rados.0.mira097.stdout:1133: finishing copy_from to mira09725123-126 2014-08-22 19:27:54,541.541 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 126 v 169 (ObjNum 161 snap 0 seq_num 161) dirty exists 2014-08-22 19:27:54,541.541 INFO:tasks.rados.rados.0.mira097.stdout:1138: write oid 51 current snap is 61 2014-08-22 19:27:54,541.541 INFO:tasks.rados.rados.0.mira097.stdout:1138: seq_num 735 ranges {405764=492223,1478670=579364,2732281=511989,3605763=1} 2014-08-22 19:27:54,557.557 INFO:tasks.rados.rados.0.mira097.stdout:1138: writing mira09725123-51 from 405764 to 897987 tid 1 2014-08-22 19:27:54,578.578 INFO:tasks.rados.rados.0.mira097.stdout:1138: writing mira09725123-51 from 1478670 to 2058034 tid 2 2014-08-22 19:27:54,598.598 INFO:tasks.rados.rados.0.mira097.stdout:1138: writing mira09725123-51 from 2732281 to 3244270 tid 3 2014-08-22 19:27:54,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1138: writing mira09725123-51 from 3605763 to 3605764 tid 4 2014-08-22 19:27:54,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1139: snap_create 2014-08-22 19:27:54,603.603 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:27:54,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1133: finishing copy_from racing read to mira09725123-126 2014-08-22 19:27:54,604.604 INFO:tasks.rados.rados.0.mira097.stdout:1133: done (5 left) 2014-08-22 19:27:54,604.604 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:27:54,734.734 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 81 v 1681 (ObjNum 80 snap 0 seq_num 80) dirty exists 2014-08-22 19:27:54,735.735 INFO:tasks.rados.rados.0.mira097.stdout:1134: done (4 left) 2014-08-22 19:27:54,735.735 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:27:54,769.769 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 431 v 1657 (ObjNum 548 snap 12 seq_num 548) dirty exists 2014-08-22 19:27:54,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1137: done (3 left) 2014-08-22 19:27:54,770.770 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:27:54,771.771 INFO:tasks.rados.rados.0.mira097.stdout:1135: expect (ObjNum 563 snap 20 seq_num 563) 2014-08-22 19:27:56,236.236 INFO:tasks.rados.rados.0.mira097.stdout:1136: finishing write tid 1 to mira09725123-153 2014-08-22 19:27:56,236.236 INFO:tasks.rados.rados.0.mira097.stdout:1136: finishing write tid 2 to mira09725123-153 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1136: finishing write tid 3 to mira09725123-153 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1136: finishing write tid 5 to mira09725123-153 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 1 to mira09725123-51 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 2 to mira09725123-51 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 3 to mira09725123-51 2014-08-22 19:27:56,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 4 to mira09725123-51 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 6 to mira09725123-51 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1136: finishing write tid 6 to mira09725123-153 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 153 v 1833 (ObjNum 734 snap 61 seq_num 734) dirty exists 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1138: finishing write tid 7 to mira09725123-51 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 51 v 1844 (ObjNum 735 snap 61 seq_num 735) dirty exists 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1135: done (2 left) 2014-08-22 19:27:56,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1136: done (1 left) 2014-08-22 19:27:56,239.239 INFO:tasks.rados.rados.0.mira097.stdout:1138: done (0 left) 2014-08-22 19:27:56,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1139: done (0 left) 2014-08-22 19:27:56,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1140: snap_remove snap 58 2014-08-22 19:27:57,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1140: done (0 left) 2014-08-22 19:27:57,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1141: write oid 404 current snap is 62 2014-08-22 19:27:57,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1141: seq_num 736 ranges {350593=1} 2014-08-22 19:27:57,911.911 INFO:tasks.rados.rados.0.mira097.stdout:1141: writing mira09725123-404 from 350593 to 350594 tid 1 2014-08-22 19:27:57,912.912 INFO:tasks.rados.rados.0.mira097.stdout:1142: read oid 360 snap 60 2014-08-22 19:27:57,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1143: snap_remove snap 52 2014-08-22 19:27:57,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1141: finishing write tid 1 to mira09725123-404 2014-08-22 19:27:57,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1141: finishing write tid 3 to mira09725123-404 2014-08-22 19:27:57,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1141: finishing write tid 4 to mira09725123-404 2014-08-22 19:27:57,954.954 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 404 v 1691 (ObjNum 736 snap 62 seq_num 736) dirty exists 2014-08-22 19:27:57,956.956 INFO:tasks.rados.rados.0.mira097.stdout:1142: expect (ObjNum 359 snap 0 seq_num 359) 2014-08-22 19:27:58,898.898 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:27:58,898.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:27:58,899.899 INFO:tasks.thrashosds.thrasher:Adding osd 5 2014-08-22 19:27:58,899.899 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-08-22 19:27:59,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.250784 7f3623981700 1 librados: starting msgr at :/0 2014-08-22 19:27:59,346.346 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.250792 7f3623981700 1 librados: starting objecter 2014-08-22 19:27:59,347.347 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.250927 7f3623981700 1 librados: setting wanted keys 2014-08-22 19:27:59,347.347 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.250933 7f3623981700 1 librados: calling monclient init 2014-08-22 19:27:59,347.347 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.251856 7f3620a2e700 10 client.?.objecter ms_handle_connect 0x7f361c0355e0 2014-08-22 19:27:59,348.348 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.251905 7f3620a2e700 10 client.?.objecter resend_mon_ops 2014-08-22 19:27:59,348.348 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253499 7f3623981700 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:27:59,348.348 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253616 7f3623981700 1 librados: init done 2014-08-22 19:27:59,349.349 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253887 7f3620a2e700 10 client.4214.objecter ms_dispatch 0x7f361c000f80 osd_map(179..179 src has 1..179) v3 2014-08-22 19:27:59,349.349 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253896 7f3620a2e700 3 client.4214.objecter handle_osd_map got epochs [179,179] > 0 2014-08-22 19:27:59,349.349 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253899 7f3620a2e700 3 client.4214.objecter handle_osd_map decoding full epoch 179 2014-08-22 19:27:59,350.350 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.253981 7f3620a2e700 20 client.4214.objecter dump_active .. 0 homeless 2014-08-22 19:27:59,350.350 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.254773 7f3620a2e700 10 client.4214.objecter ms_dispatch 0x7f361c000f80 osd_map(179..179 src has 1..179) v3 2014-08-22 19:27:59,450.450 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.254779 7f3620a2e700 3 client.4214.objecter handle_osd_map ignoring epochs [179,179] <= 179 2014-08-22 19:27:59,451.451 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.254782 7f3620a2e700 20 client.4214.objecter dump_active .. 0 homeless 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1141: done (2 left) 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1142: done (1 left) 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1143: done (0 left) 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1144: read oid 465 snap -1 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1145: snap_create 2014-08-22 19:27:59,451.451 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:27:59,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1144: expect (ObjNum 464 snap 0 seq_num 464) 2014-08-22 19:27:59,934.934 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.5. 2014-08-22 19:27:59,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.858938 7f3623981700 20 client.4214.objecter shutdown clearing up homeless session... 2014-08-22 19:27:59,955.955 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:27:52.860316 7f3623981700 1 librados: shutdown 2014-08-22 19:27:59,967.967 INFO:tasks.thrashosds.thrasher:Added osd 5 2014-08-22 19:28:00,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1144: done (0 left) 2014-08-22 19:28:00,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1145: done (0 left) 2014-08-22 19:28:00,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1146: read oid 166 snap -1 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1147: write oid 356 current snap is 63 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1147: seq_num 737 ranges {45823=1} 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1147: writing mira09725123-356 from 45823 to 45824 tid 1 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1148: rollback oid 266 current snap is 63 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 266 to 62 2014-08-22 19:28:00,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1149: rollback oid 65 current snap is 63 2014-08-22 19:28:00,950.950 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 65 to 60 2014-08-22 19:28:00,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1150: write oid 22 current snap is 63 2014-08-22 19:28:00,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1150: seq_num 738 ranges {790653=1} 2014-08-22 19:28:00,953.953 INFO:tasks.rados.rados.0.mira097.stdout:1150: writing mira09725123-22 from 790653 to 790654 tid 1 2014-08-22 19:28:00,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1151: read oid 459 snap -1 2014-08-22 19:28:00,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1152: write oid 13 current snap is 63 2014-08-22 19:28:00,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1152: seq_num 739 ranges {734459=567985,1534458=1} 2014-08-22 19:28:00,980.980 INFO:tasks.rados.rados.0.mira097.stdout:1152: writing mira09725123-13 from 734459 to 1302444 tid 1 2014-08-22 19:28:00,982.982 INFO:tasks.rados.rados.0.mira097.stdout:1152: writing mira09725123-13 from 1534458 to 1534459 tid 2 2014-08-22 19:28:00,983.983 INFO:tasks.rados.rados.0.mira097.stdout:1153: write oid 194 current snap is 63 2014-08-22 19:28:00,983.983 INFO:tasks.rados.rados.0.mira097.stdout:1153: seq_num 740 ranges {547065=788365,1747064=1} 2014-08-22 19:28:01,008.008 INFO:tasks.rados.rados.0.mira097.stdout:1153: writing mira09725123-194 from 547065 to 1335430 tid 1 2014-08-22 19:28:01,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1153: writing mira09725123-194 from 1747064 to 1747065 tid 2 2014-08-22 19:28:01,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1154: copy_from oid 36 from oid 208 current snap is 63 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1155: write oid 139 current snap is 63 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1155: seq_num 741 ranges {491895=795241,1789448=635607,2491894=1} 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1155: writing mira09725123-139 from 491895 to 1287136 tid 1 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1155: writing mira09725123-139 from 1789448 to 2425055 tid 2 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1155: writing mira09725123-139 from 2491894 to 2491895 tid 3 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1156: snap_create 2014-08-22 19:28:01,116.116 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 10 2014-08-22 19:28:01,266.266 INFO:tasks.rados.rados.0.mira097.stdout:1146: expect (ObjNum 260 snap 0 seq_num 260) 2014-08-22 19:28:01,924.924 INFO:tasks.rados.rados.0.mira097.stdout:1147: finishing write tid 1 to mira09725123-356 2014-08-22 19:28:01,924.924 INFO:tasks.rados.rados.0.mira097.stdout:1147: finishing write tid 3 to mira09725123-356 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1147: finishing write tid 4 to mira09725123-356 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 356 v 1675 (ObjNum 737 snap 63 seq_num 737) dirty exists 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1153: finishing write tid 1 to mira09725123-194 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1153: finishing write tid 2 to mira09725123-194 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1153: finishing write tid 4 to mira09725123-194 2014-08-22 19:28:01,925.925 INFO:tasks.rados.rados.0.mira097.stdout:1152: finishing write tid 1 to mira09725123-13 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:1152: finishing write tid 2 to mira09725123-13 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:1152: finishing write tid 4 to mira09725123-13 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 266 v 1715 (ObjNum 265 snap 0 seq_num 265) dirty exists 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:1153: finishing write tid 5 to mira09725123-194 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 194 v 1682 (ObjNum 740 snap 63 seq_num 740) dirty exists 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:1154: finishing copy_from to mira09725123-36 2014-08-22 19:28:01,926.926 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 36 v 241 (ObjNum 207 snap 0 seq_num 207) dirty exists 2014-08-22 19:28:01,927.927 INFO:tasks.rados.rados.0.mira097.stdout:1151: expect (ObjNum 458 snap 0 seq_num 458) 2014-08-22 19:28:01,937.937 INFO:tasks.rados.rados.0.mira097.stdout:1152: finishing write tid 5 to mira09725123-13 2014-08-22 19:28:01,937.937 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 13 v 1713 (ObjNum 739 snap 63 seq_num 739) dirty exists 2014-08-22 19:28:01,937.937 INFO:tasks.rados.rados.0.mira097.stdout:1154: finishing copy_from racing read to mira09725123-36 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1155: finishing write tid 1 to mira09725123-139 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1155: finishing write tid 2 to mira09725123-139 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1155: finishing write tid 3 to mira09725123-139 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1155: finishing write tid 5 to mira09725123-139 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1146: done (9 left) 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1147: done (8 left) 2014-08-22 19:28:01,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1148: done (7 left) 2014-08-22 19:28:01,939.939 INFO:tasks.rados.rados.0.mira097.stdout:1151: done (6 left) 2014-08-22 19:28:01,939.939 INFO:tasks.rados.rados.0.mira097.stdout:1152: done (5 left) 2014-08-22 19:28:01,939.939 INFO:tasks.rados.rados.0.mira097.stdout:1153: done (4 left) 2014-08-22 19:28:01,939.939 INFO:tasks.rados.rados.0.mira097.stdout:1154: done (3 left) 2014-08-22 19:28:01,939.939 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:28:03,723.723 INFO:tasks.rados.rados.0.mira097.stdout:1155: finishing write tid 6 to mira09725123-139 2014-08-22 19:28:03,723.723 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 139 v 1720 (ObjNum 741 snap 63 seq_num 741) dirty exists 2014-08-22 19:28:03,724.724 INFO:tasks.rados.rados.0.mira097.stdout:1155: done (2 left) 2014-08-22 19:28:03,724.724 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:04,183.183 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 65 v 1593 (ObjNum 64 snap 0 seq_num 64) dirty exists 2014-08-22 19:28:04,184.184 INFO:tasks.rados.rados.0.mira097.stdout:1149: done (1 left) 2014-08-22 19:28:04,184.184 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:04,234.234 INFO:tasks.rados.rados.0.mira097.stdout:1150: finishing write tid 1 to mira09725123-22 2014-08-22 19:28:04,239.239 INFO:tasks.rados.rados.0.mira097.stdout:1150: finishing write tid 3 to mira09725123-22 2014-08-22 19:28:04,244.244 INFO:tasks.rados.rados.0.mira097.stdout:1150: finishing write tid 4 to mira09725123-22 2014-08-22 19:28:04,244.244 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 22 v 1597 (ObjNum 738 snap 63 seq_num 738) dirty exists 2014-08-22 19:28:04,244.244 INFO:tasks.rados.rados.0.mira097.stdout:1150: done (0 left) 2014-08-22 19:28:04,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1156: done (0 left) 2014-08-22 19:28:04,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1157: rollback oid 158 current snap is 64 2014-08-22 19:28:04,538.538 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 158 to 60 2014-08-22 19:28:04,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1158: write oid 241 current snap is 64 2014-08-22 19:28:04,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1158: seq_num 742 ranges {435701=767046,1999709=457226,3116988=118713} 2014-08-22 19:28:04,569.569 INFO:tasks.rados.rados.0.mira097.stdout:1158: writing mira09725123-241 from 435701 to 1202747 tid 1 2014-08-22 19:28:04,588.588 INFO:tasks.rados.rados.0.mira097.stdout:1158: writing mira09725123-241 from 1999709 to 2456935 tid 2 2014-08-22 19:28:04,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1158: writing mira09725123-241 from 3116988 to 3235701 tid 3 2014-08-22 19:28:04,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1159: copy_from oid 93 from oid 99 current snap is 64 2014-08-22 19:28:04,596.596 INFO:tasks.rados.rados.0.mira097.stdout:1160: copy_from oid 63 from oid 238 current snap is 64 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1161: write oid 332 current snap is 64 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1161: seq_num 743 ranges {531954=491298,1466288=678845,2822084=109870} 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1161: writing mira09725123-332 from 531954 to 1023252 tid 1 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1161: writing mira09725123-332 from 1466288 to 2145133 tid 2 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1161: writing mira09725123-332 from 2822084 to 2931954 tid 3 2014-08-22 19:28:04,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1162: copy_from oid 27 from oid 92 current snap is 64 2014-08-22 19:28:04,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1163: snap_create 2014-08-22 19:28:04,715.715 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:28:04,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1161: finishing write tid 1 to mira09725123-332 2014-08-22 19:28:04,912.912 INFO:tasks.rados.rados.0.mira097.stdout:1161: finishing write tid 2 to mira09725123-332 2014-08-22 19:28:04,915.915 INFO:tasks.rados.rados.0.mira097.stdout:1161: finishing write tid 3 to mira09725123-332 2014-08-22 19:28:04,968.968 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:04,968.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:04,969.969 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2014-08-22 19:28:04,969.969 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:28:04,969.969 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:28:04,970.970 INFO:teuthology.orchestra.run.mira097:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-08-22 19:28:08,982.982 INFO:tasks.rados.rados.0.mira097.stdout:1158: finishing write tid 1 to mira09725123-241 2014-08-22 19:28:08,982.982 INFO:tasks.rados.rados.0.mira097.stdout:1158: finishing write tid 2 to mira09725123-241 2014-08-22 19:28:08,982.982 INFO:tasks.rados.rados.0.mira097.stdout:1158: finishing write tid 3 to mira09725123-241 2014-08-22 19:28:08,982.982 INFO:tasks.rados.rados.0.mira097.stdout:1158: finishing write tid 5 to mira09725123-241 2014-08-22 19:28:08,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1158: finishing write tid 6 to mira09725123-241 2014-08-22 19:28:08,984.984 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 241 v 1603 (ObjNum 742 snap 64 seq_num 742) dirty exists 2014-08-22 19:28:08,984.984 INFO:tasks.rados.rados.0.mira097.stdout:1158: done (5 left) 2014-08-22 19:28:08,984.984 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:28:09,010.010 INFO:tasks.rados.rados.0.mira097.stdout:1161: finishing write tid 5 to mira09725123-332 2014-08-22 19:28:09,011.011 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 158 v 1606 (ObjNum 20 snap 0 seq_num 20) dirty exists 2014-08-22 19:28:09,160.160 INFO:tasks.rados.rados.0.mira097.stdout:1157: done (4 left) 2014-08-22 19:28:09,160.160 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:28:09,160.160 INFO:tasks.rados.rados.0.mira097.stdout:1162: finishing copy_from to mira09725123-27 2014-08-22 19:28:09,160.160 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 27 v 124 (ObjNum 91 snap 0 seq_num 91) dirty exists 2014-08-22 19:28:09,160.160 INFO:tasks.rados.rados.0.mira097.stdout:1161: finishing write tid 6 to mira09725123-332 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 332 v 1726 (ObjNum 743 snap 64 seq_num 743) dirty exists 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1161: done (3 left) 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1162: finishing copy_from racing read to mira09725123-27 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1162: done (2 left) 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1160: finishing copy_from to mira09725123-63 2014-08-22 19:28:09,161.161 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 63 v 325 (ObjNum 237 snap 0 seq_num 237) dirty exists 2014-08-22 19:28:09,162.162 INFO:tasks.rados.rados.0.mira097.stdout:1160: finishing copy_from racing read to mira09725123-63 2014-08-22 19:28:09,162.162 INFO:tasks.rados.rados.0.mira097.stdout:1160: done (1 left) 2014-08-22 19:28:09,162.162 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:09,354.354 INFO:tasks.rados.rados.0.mira097.stdout:1159: finishing copy_from to mira09725123-93 2014-08-22 19:28:09,354.354 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 93 v 96 (ObjNum 98 snap 0 seq_num 98) dirty exists 2014-08-22 19:28:09,355.355 INFO:tasks.rados.rados.0.mira097.stdout:1159: finishing copy_from racing read to mira09725123-93 2014-08-22 19:28:09,355.355 INFO:tasks.rados.rados.0.mira097.stdout:1159: done (0 left) 2014-08-22 19:28:09,810.810 INFO:tasks.rados.rados.0.mira097.stdout:1163: done (0 left) 2014-08-22 19:28:09,810.810 INFO:tasks.rados.rados.0.mira097.stdout:1164: snap_create 2014-08-22 19:28:10,438.438 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:10,439.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:10,439.439 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.798220639821 2014-08-22 19:28:10,439.439 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 0.798220639821' 2014-08-22 19:28:10,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1164: done (0 left) 2014-08-22 19:28:10,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1165: read oid 13 snap -1 2014-08-22 19:28:10,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1166: write oid 453 current snap is 66 2014-08-22 19:28:10,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1166: seq_num 744 ranges {191087=1} 2014-08-22 19:28:10,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1166: writing mira09725123-453 from 191087 to 191088 tid 1 2014-08-22 19:28:10,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1167: copy_from oid 76 from oid 439 current snap is 66 2014-08-22 19:28:10,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1168: read oid 286 snap -1 2014-08-22 19:28:10,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1169: rollback oid 301 current snap is 66 2014-08-22 19:28:10,841.841 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 301 to 64 2014-08-22 19:28:10,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1170: copy_from oid 436 from oid 488 current snap is 66 2014-08-22 19:28:10,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1171: read oid 452 snap 62 2014-08-22 19:28:10,845.845 INFO:tasks.rados.rados.0.mira097.stdout:1172: write oid 42 current snap is 66 2014-08-22 19:28:10,845.845 INFO:tasks.rados.rados.0.mira097.stdout:1172: seq_num 745 ranges {535918=400000} 2014-08-22 19:28:10,865.865 INFO:tasks.rados.rados.0.mira097.stdout:1172: writing mira09725123-42 from 535918 to 935918 tid 1 2014-08-22 19:28:10,866.866 INFO:tasks.rados.rados.0.mira097.stdout:1173: rollback oid 4 current snap is 66 2014-08-22 19:28:10,866.866 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 4 to 60 2014-08-22 19:28:11,014.014 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 301 v 0 (ObjNum 29880840 snap 0 seq_num 10206096) dirty dne 2014-08-22 19:28:11,014.014 INFO:tasks.rados.rados.0.mira097.stdout:1169: done (8 left) 2014-08-22 19:28:11,015.015 INFO:tasks.rados.rados.0.mira097.stdout:1174: snap_remove snap 61 2014-08-22 19:28:11,015.015 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.834545 7fe53a174700 1 librados: starting msgr at :/0 2014-08-22 19:28:11,015.015 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.834549 7fe53a174700 1 librados: starting objecter 2014-08-22 19:28:11,015.015 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.834719 7fe53a174700 1 librados: setting wanted keys 2014-08-22 19:28:11,015.015 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.834726 7fe53a174700 1 librados: calling monclient init 2014-08-22 19:28:11,015.015 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.835829 7fe532ffd700 10 client.?.objecter ms_handle_connect 0x7fe5340355e0 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.835882 7fe532ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.839244 7fe53a174700 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.839406 7fe53a174700 1 librados: init done 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840236 7fe532ffd700 10 client.4224.objecter ms_dispatch 0x7fe534000f80 osd_map(184..184 src has 1..184) v3 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840247 7fe532ffd700 3 client.4224.objecter handle_osd_map got epochs [184,184] > 0 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840251 7fe532ffd700 3 client.4224.objecter handle_osd_map decoding full epoch 184 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840366 7fe532ffd700 20 client.4224.objecter dump_active .. 0 homeless 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840422 7fe532ffd700 10 client.4224.objecter ms_dispatch 0x7fe534000f80 osd_map(184..184 src has 1..184) v3 2014-08-22 19:28:11,016.016 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840430 7fe532ffd700 3 client.4224.objecter handle_osd_map ignoring epochs [184,184] <= 184 2014-08-22 19:28:11,017.017 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:03.840434 7fe532ffd700 20 client.4224.objecter dump_active .. 0 homeless 2014-08-22 19:28:11,850.850 INFO:tasks.rados.rados.0.mira097.stdout:1174: done (8 left) 2014-08-22 19:28:11,850.850 INFO:tasks.rados.rados.0.mira097.stdout:1175: rollback oid 334 current snap is 66 2014-08-22 19:28:11,850.850 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 334 to 65 2014-08-22 19:28:11,850.850 INFO:tasks.rados.rados.0.mira097.stdout:1176: snap_remove snap 63 2014-08-22 19:28:11,882.882 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 0.798221 (8523122) 2014-08-22 19:28:11,886.886 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:04.791008 7fe53a174700 20 client.4224.objecter shutdown clearing up homeless session... 2014-08-22 19:28:11,886.886 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:04.791398 7fe53a174700 1 librados: shutdown 2014-08-22 19:28:12,854.854 INFO:tasks.rados.rados.0.mira097.stdout:1176: done (9 left) 2014-08-22 19:28:12,855.855 INFO:tasks.rados.rados.0.mira097.stdout:1177: write oid 388 current snap is 66 2014-08-22 19:28:12,855.855 INFO:tasks.rados.rados.0.mira097.stdout:1177: seq_num 746 ranges {632171=1} 2014-08-22 19:28:12,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1177: writing mira09725123-388 from 632171 to 632172 tid 1 2014-08-22 19:28:12,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1178: snap_remove snap 64 2014-08-22 19:28:13,862.862 INFO:tasks.rados.rados.0.mira097.stdout:1178: done (10 left) 2014-08-22 19:28:13,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1179: rollback oid 475 current snap is 66 2014-08-22 19:28:13,863.863 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 475 to 62 2014-08-22 19:28:13,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1180: read oid 64 snap -1 2014-08-22 19:28:13,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1181: rollback oid 137 current snap is 66 2014-08-22 19:28:13,864.864 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 137 to 60 2014-08-22 19:28:13,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1182: copy_from oid 355 from oid 430 current snap is 66 2014-08-22 19:28:13,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1183: delete oid 351 current snap is 66 2014-08-22 19:28:14,090.090 INFO:tasks.rados.rados.0.mira097.stdout:1170: finishing copy_from to mira09725123-436 2014-08-22 19:28:14,091.091 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 436 v 596 (ObjNum 487 snap 0 seq_num 487) dirty exists 2014-08-22 19:28:14,103.103 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 137 v 0 (ObjNum 18995296 snap 0 seq_num 0) dirty dne 2014-08-22 19:28:14,107.107 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 334 v 1874 (ObjNum 333 snap 0 seq_num 333) dirty exists 2014-08-22 19:28:14,114.114 INFO:tasks.rados.rados.0.mira097.stdout:1171: expect (ObjNum 451 snap 0 seq_num 451) 2014-08-22 19:28:14,158.158 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 4 v 1689 (ObjNum 3 snap 0 seq_num 3) dirty exists 2014-08-22 19:28:14,158.158 INFO:tasks.rados.rados.0.mira097.stdout:1180: expect (ObjNum 631 snap 36 seq_num 631) 2014-08-22 19:28:14,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1177: finishing write tid 1 to mira09725123-388 2014-08-22 19:28:15,022.022 INFO:tasks.rados.rados.0.mira097.stdout:1177: finishing write tid 3 to mira09725123-388 2014-08-22 19:28:15,023.023 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 475 v 1881 (ObjNum 474 snap 0 seq_num 474) dirty exists 2014-08-22 19:28:15,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1171: done (14 left) 2014-08-22 19:28:15,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1173: done (13 left) 2014-08-22 19:28:15,025.025 INFO:tasks.rados.rados.0.mira097.stdout:1175: done (12 left) 2014-08-22 19:28:15,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1179: done (11 left) 2014-08-22 19:28:15,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1180: done (10 left) 2014-08-22 19:28:15,027.027 INFO:tasks.rados.rados.0.mira097.stdout:1181: done (9 left) 2014-08-22 19:28:15,027.027 INFO:tasks.rados.rados.0.mira097.stdout:1183: done (8 left) 2014-08-22 19:28:15,028.028 INFO:tasks.rados.rados.0.mira097.stdout:1184: rollback oid 6 current snap is 66 2014-08-22 19:28:15,028.028 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 6 to 65 2014-08-22 19:28:15,028.028 INFO:tasks.rados.rados.0.mira097.stdout:1185: read oid 418 snap -1 2014-08-22 19:28:15,028.028 INFO:tasks.rados.rados.0.mira097.stdout:1186: read oid 210 snap -1 2014-08-22 19:28:15,031.031 INFO:tasks.rados.rados.0.mira097.stdout:1187: write oid 471 current snap is 66 2014-08-22 19:28:15,032.032 INFO:tasks.rados.rados.0.mira097.stdout:1187: seq_num 748 ranges {520807=537840,1812309=308498} 2014-08-22 19:28:15,055.055 INFO:tasks.rados.rados.0.mira097.stdout:1187: writing mira09725123-471 from 520807 to 1058647 tid 1 2014-08-22 19:28:15,068.068 INFO:tasks.rados.rados.0.mira097.stdout:1187: writing mira09725123-471 from 1812309 to 2120807 tid 2 2014-08-22 19:28:15,073.073 INFO:tasks.rados.rados.0.mira097.stdout:1188: snap_remove snap 62 2014-08-22 19:28:15,074.074 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 6 v 1194 (ObjNum 1 snap 1 seq_num 8617712) dirty dne 2014-08-22 19:28:15,078.078 INFO:tasks.rados.rados.0.mira097.stdout:1165: expect (ObjNum 739 snap 63 seq_num 739) 2014-08-22 19:28:15,487.487 INFO:tasks.rados.rados.0.mira097.stdout:1185: expect (ObjNum 40 snap 0 seq_num 40) 2014-08-22 19:28:16,456.456 INFO:tasks.rados.rados.0.mira097.stdout:1168: expect (ObjNum 579 snap 23 seq_num 579) 2014-08-22 19:28:16,892.892 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:16,892.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:16,893.893 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2014-08-22 19:28:16,893.893 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:28:16,893.893 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:28:16,894.894 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-08-22 19:28:17,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1170: finishing copy_from racing read to mira09725123-436 2014-08-22 19:28:17,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1187: finishing write tid 1 to mira09725123-471 2014-08-22 19:28:17,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1187: finishing write tid 2 to mira09725123-471 2014-08-22 19:28:17,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1187: finishing write tid 4 to mira09725123-471 2014-08-22 19:28:17,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1165: done (12 left) 2014-08-22 19:28:17,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1168: done (11 left) 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1170: done (10 left) 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1184: done (9 left) 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1185: done (8 left) 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1188: done (7 left) 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1189: read oid 225 snap -1 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:1177: finishing write tid 4 to mira09725123-388 2014-08-22 19:28:17,453.453 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 388 v 1751 (ObjNum 746 snap 66 seq_num 746) dirty exists 2014-08-22 19:28:17,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1182: finishing copy_from to mira09725123-355 2014-08-22 19:28:17,454.454 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 355 v 1640 (ObjNum 490 snap 0 seq_num 490) dirty exists 2014-08-22 19:28:17,454.454 INFO:tasks.rados.rados.0.mira097.stdout:1186: expect (ObjNum 209 snap 0 seq_num 209) 2014-08-22 19:28:18,141.141 INFO:tasks.rados.rados.0.mira097.stdout:1167: finishing copy_from to mira09725123-76 2014-08-22 19:28:18,141.141 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 76 v 610 (ObjNum 438 snap 0 seq_num 438) dirty exists 2014-08-22 19:28:18,141.141 INFO:tasks.rados.rados.0.mira097.stdout:1187: finishing write tid 5 to mira09725123-471 2014-08-22 19:28:18,141.141 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 471 v 1893 (ObjNum 748 snap 66 seq_num 748) dirty exists 2014-08-22 19:28:18,142.142 INFO:tasks.rados.rados.0.mira097.stdout:1166: finishing write tid 1 to mira09725123-453 2014-08-22 19:28:18,142.142 INFO:tasks.rados.rados.0.mira097.stdout:1166: finishing write tid 3 to mira09725123-453 2014-08-22 19:28:18,142.142 INFO:tasks.rados.rados.0.mira097.stdout:1166: finishing write tid 4 to mira09725123-453 2014-08-22 19:28:18,142.142 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 453 v 1626 (ObjNum 744 snap 66 seq_num 744) dirty exists 2014-08-22 19:28:18,142.142 INFO:tasks.rados.rados.0.mira097.stdout:1182: finishing copy_from racing read to mira09725123-355 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1166: done (7 left) 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1177: done (6 left) 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1182: done (5 left) 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1186: done (4 left) 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1187: done (3 left) 2014-08-22 19:28:18,143.143 INFO:tasks.rados.rados.0.mira097.stdout:1190: read oid 155 snap -1 2014-08-22 19:28:18,144.144 INFO:tasks.rados.rados.0.mira097.stdout:1167: finishing copy_from racing read to mira09725123-76 2014-08-22 19:28:18,144.144 INFO:tasks.rados.rados.0.mira097.stdout:1172: finishing write tid 1 to mira09725123-42 2014-08-22 19:28:18,144.144 INFO:tasks.rados.rados.0.mira097.stdout:1172: finishing write tid 3 to mira09725123-42 2014-08-22 19:28:18,144.144 INFO:tasks.rados.rados.0.mira097.stdout:1172: finishing write tid 4 to mira09725123-42 2014-08-22 19:28:18,145.145 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 42 v 1637 (ObjNum 745 snap 66 seq_num 745) dirty exists 2014-08-22 19:28:18,145.145 INFO:tasks.rados.rados.0.mira097.stdout:1189: expect (ObjNum 642 snap 42 seq_num 642) 2014-08-22 19:28:18,745.745 INFO:tasks.rados.rados.0.mira097.stdout:1190: expect (ObjNum 573 snap 23 seq_num 573) 2014-08-22 19:28:18,761.761 INFO:tasks.rados.rados.0.mira097.stdout:1167: done (3 left) 2014-08-22 19:28:18,761.761 INFO:tasks.rados.rados.0.mira097.stdout:1172: done (2 left) 2014-08-22 19:28:18,762.762 INFO:tasks.rados.rados.0.mira097.stdout:1189: done (1 left) 2014-08-22 19:28:18,762.762 INFO:tasks.rados.rados.0.mira097.stdout:1190: done (0 left) 2014-08-22 19:28:18,762.762 INFO:tasks.rados.rados.0.mira097.stdout:1191: read oid 467 snap 65 2014-08-22 19:28:18,762.762 INFO:tasks.rados.rados.0.mira097.stdout:1192: copy_from oid 86 from oid 316 current snap is 66 2014-08-22 19:28:18,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1193: write oid 254 current snap is 66 2014-08-22 19:28:18,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1193: seq_num 749 ranges {732389=544716,2041769=290620} 2014-08-22 19:28:18,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1193: writing mira09725123-254 from 732389 to 1277105 tid 1 2014-08-22 19:28:18,804.804 INFO:tasks.rados.rados.0.mira097.stdout:1193: writing mira09725123-254 from 2041769 to 2332389 tid 2 2014-08-22 19:28:18,804.804 INFO:tasks.rados.rados.0.mira097.stdout:1194: snap_create 2014-08-22 19:28:18,804.804 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:28:20,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1191: expect (ObjNum 466 snap 0 seq_num 466) 2014-08-22 19:28:20,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1191: done (2 left) 2014-08-22 19:28:20,497.497 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:21,120.120 INFO:tasks.rados.rados.0.mira097.stdout:1192: finishing copy_from to mira09725123-86 2014-08-22 19:28:21,120.120 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 86 v 1075 (ObjNum 613 snap 29 seq_num 613) dirty exists 2014-08-22 19:28:21,129.129 INFO:tasks.rados.rados.0.mira097.stdout:1192: finishing copy_from racing read to mira09725123-86 2014-08-22 19:28:21,132.132 INFO:tasks.rados.rados.0.mira097.stdout:1192: done (1 left) 2014-08-22 19:28:21,132.132 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:21,183.183 INFO:tasks.rados.rados.0.mira097.stdout:1193: finishing write tid 1 to mira09725123-254 2014-08-22 19:28:21,199.199 INFO:tasks.rados.rados.0.mira097.stdout:1193: finishing write tid 2 to mira09725123-254 2014-08-22 19:28:21,202.202 INFO:tasks.rados.rados.0.mira097.stdout:1193: finishing write tid 4 to mira09725123-254 2014-08-22 19:28:21,202.202 INFO:tasks.rados.rados.0.mira097.stdout:1193: finishing write tid 5 to mira09725123-254 2014-08-22 19:28:21,202.202 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 254 v 1644 (ObjNum 749 snap 66 seq_num 749) dirty exists 2014-08-22 19:28:21,202.202 INFO:tasks.rados.rados.0.mira097.stdout:1193: done (0 left) 2014-08-22 19:28:22,333.333 INFO:tasks.rados.rados.0.mira097.stdout:1194: done (0 left) 2014-08-22 19:28:22,337.337 INFO:tasks.rados.rados.0.mira097.stdout:1195: delete oid 371 current snap is 67 2014-08-22 19:28:22,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1195: done (0 left) 2014-08-22 19:28:22,367.367 INFO:tasks.rados.rados.0.mira097.stdout:1196: write oid 66 current snap is 67 2014-08-22 19:28:22,367.367 INFO:tasks.rados.rados.0.mira097.stdout:1196: seq_num 751 ranges {622048=489349,1684561=626903,2874768=657758,3822047=1} 2014-08-22 19:28:22,388.388 INFO:tasks.rados.rados.0.mira097.stdout:1196: writing mira09725123-66 from 622048 to 1111397 tid 1 2014-08-22 19:28:22,400.400 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:22,401.401 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:28:22,407.407 INFO:tasks.rados.rados.0.mira097.stdout:1196: writing mira09725123-66 from 1684561 to 2311464 tid 2 2014-08-22 19:28:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1196: writing mira09725123-66 from 2874768 to 3532526 tid 3 2014-08-22 19:28:22,439.439 INFO:tasks.rados.rados.0.mira097.stdout:1196: writing mira09725123-66 from 3822047 to 3822048 tid 4 2014-08-22 19:28:22,439.439 INFO:tasks.rados.rados.0.mira097.stdout:1197: copy_from oid 182 from oid 23 current snap is 67 2014-08-22 19:28:22,440.440 INFO:tasks.rados.rados.0.mira097.stdout:1198: write oid 368 current snap is 67 2014-08-22 19:28:22,440.440 INFO:tasks.rados.rados.0.mira097.stdout:1198: seq_num 752 ranges {716254=461153,1645221=696074,2922521=593733} 2014-08-22 19:28:22,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1198: writing mira09725123-368 from 716254 to 1177407 tid 1 2014-08-22 19:28:22,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1198: writing mira09725123-368 from 1645221 to 2341295 tid 2 2014-08-22 19:28:22,510.510 INFO:tasks.rados.rados.0.mira097.stdout:1198: writing mira09725123-368 from 2922521 to 3516254 tid 3 2014-08-22 19:28:22,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1199: read oid 137 snap -1 2014-08-22 19:28:22,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1200: copy_from oid 304 from oid 130 current snap is 67 2014-08-22 19:28:22,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1201: snap_remove snap 66 2014-08-22 19:28:22,788.788 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 1 to mira09725123-66 2014-08-22 19:28:22,788.788 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 2 to mira09725123-66 2014-08-22 19:28:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 3 to mira09725123-66 2014-08-22 19:28:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 4 to mira09725123-66 2014-08-22 19:28:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 6 to mira09725123-66 2014-08-22 19:28:22,875.875 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.778544 7ffed8a02700 1 librados: starting msgr at :/0 2014-08-22 19:28:22,875.875 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.778556 7ffed8a02700 1 librados: starting objecter 2014-08-22 19:28:22,875.875 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.778848 7ffed8a02700 1 librados: setting wanted keys 2014-08-22 19:28:22,875.875 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.778862 7ffed8a02700 1 librados: calling monclient init 2014-08-22 19:28:22,876.876 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.781287 7ffed18ad700 10 client.?.objecter ms_handle_connect 0x7ffed40355e0 2014-08-22 19:28:22,877.877 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.781337 7ffed18ad700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:22,880.880 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.784388 7ffed8a02700 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:22,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.784573 7ffed8a02700 1 librados: init done 2014-08-22 19:28:22,953.953 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785375 7ffed18ad700 10 client.4225.objecter ms_dispatch 0x7ffed4000f80 osd_map(189..189 src has 1..189) v3 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785388 7ffed18ad700 3 client.4225.objecter handle_osd_map got epochs [189,189] > 0 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785392 7ffed18ad700 3 client.4225.objecter handle_osd_map decoding full epoch 189 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785552 7ffed18ad700 20 client.4225.objecter dump_active .. 0 homeless 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785612 7ffed18ad700 10 client.4225.objecter ms_dispatch 0x7ffed4000f80 osd_map(189..189 src has 1..189) v3 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785617 7ffed18ad700 3 client.4225.objecter handle_osd_map ignoring epochs [189,189] <= 189 2014-08-22 19:28:22,954.954 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:15.785620 7ffed18ad700 20 client.4225.objecter dump_active .. 0 homeless 2014-08-22 19:28:23,124.124 INFO:tasks.rados.rados.0.mira097.stdout:1198: finishing write tid 1 to mira09725123-368 2014-08-22 19:28:23,125.125 INFO:tasks.rados.rados.0.mira097.stdout:1198: finishing write tid 2 to mira09725123-368 2014-08-22 19:28:23,126.126 INFO:tasks.rados.rados.0.mira097.stdout:1198: finishing write tid 3 to mira09725123-368 2014-08-22 19:28:23,127.127 INFO:tasks.rados.rados.0.mira097.stdout:1198: finishing write tid 5 to mira09725123-368 2014-08-22 19:28:23,133.133 INFO:tasks.rados.rados.0.mira097.stdout:1198: finishing write tid 6 to mira09725123-368 2014-08-22 19:28:23,133.133 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 368 v 1699 (ObjNum 752 snap 67 seq_num 752) dirty exists 2014-08-22 19:28:23,140.140 INFO:tasks.rados.rados.0.mira097.stdout:1196: finishing write tid 7 to mira09725123-66 2014-08-22 19:28:23,140.140 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 66 v 1926 (ObjNum 751 snap 67 seq_num 751) dirty exists 2014-08-22 19:28:23,187.187 INFO:tasks.rados.rados.0.mira097.stdout:1197: finishing copy_from to mira09725123-182 2014-08-22 19:28:23,187.187 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 182 v 27 (ObjNum 22 snap 0 seq_num 22) dirty exists 2014-08-22 19:28:23,225.225 INFO:tasks.rados.rados.0.mira097.stdout:1200: finishing copy_from to mira09725123-304 2014-08-22 19:28:23,226.226 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 304 v 168 (ObjNum 129 snap 0 seq_num 129) dirty exists 2014-08-22 19:28:23,715.715 INFO:tasks.rados.rados.0.mira097.stdout:1197: finishing copy_from racing read to mira09725123-182 2014-08-22 19:28:23,718.718 INFO:tasks.rados.rados.0.mira097.stdout:1196: done (5 left) 2014-08-22 19:28:23,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1197: done (4 left) 2014-08-22 19:28:23,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1198: done (3 left) 2014-08-22 19:28:23,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1199: done (2 left) 2014-08-22 19:28:23,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1201: done (1 left) 2014-08-22 19:28:23,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1202: snap_remove snap 65 2014-08-22 19:28:23,737.737 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:28:23,739.739 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:16.641451 7ffed8a02700 20 client.4225.objecter shutdown clearing up homeless session... 2014-08-22 19:28:23,741.741 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:16.642081 7ffed8a02700 1 librados: shutdown 2014-08-22 19:28:23,752.752 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:28:24,203.203 INFO:tasks.rados.rados.0.mira097.stdout:1200: finishing copy_from racing read to mira09725123-304 2014-08-22 19:28:24,401.401 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.302665 7f4abe53c700 1 librados: starting msgr at :/0 2014-08-22 19:28:24,401.401 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.302675 7f4abe53c700 1 librados: starting objecter 2014-08-22 19:28:24,408.408 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.302992 7f4abe53c700 1 librados: setting wanted keys 2014-08-22 19:28:24,408.408 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.303004 7f4abe53c700 1 librados: calling monclient init 2014-08-22 19:28:24,409.409 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.304395 7f4aaffff700 10 client.?.objecter ms_handle_connect 0x7f4ab002e840 2014-08-22 19:28:24,409.409 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.304439 7f4aaffff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:24,414.414 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.307523 7f4abe53c700 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:24,414.414 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.307801 7f4abe53c700 1 librados: init done 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308293 7f4aaffff700 10 client.4227.objecter ms_dispatch 0x7f4ab8000f80 osd_map(190..190 src has 1..190) v3 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308308 7f4aaffff700 3 client.4227.objecter handle_osd_map got epochs [190,190] > 0 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308311 7f4aaffff700 3 client.4227.objecter handle_osd_map decoding full epoch 190 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308434 7f4aaffff700 20 client.4227.objecter dump_active .. 0 homeless 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308482 7f4aaffff700 10 client.4227.objecter ms_dispatch 0x7f4ab8000f80 osd_map(190..190 src has 1..190) v3 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308488 7f4aaffff700 3 client.4227.objecter handle_osd_map ignoring epochs [190,190] <= 190 2014-08-22 19:28:24,415.415 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.308491 7f4aaffff700 20 client.4227.objecter dump_active .. 0 homeless 2014-08-22 19:28:24,886.886 INFO:tasks.rados.rados.0.mira097.stdout:1200: done (1 left) 2014-08-22 19:28:24,886.886 INFO:tasks.rados.rados.0.mira097.stdout:1202: done (0 left) 2014-08-22 19:28:24,886.886 INFO:tasks.rados.rados.0.mira097.stdout:1203: read oid 108 snap 60 2014-08-22 19:28:24,887.887 INFO:tasks.rados.rados.0.mira097.stdout:1204: write oid 265 current snap is 67 2014-08-22 19:28:24,887.887 INFO:tasks.rados.rados.0.mira097.stdout:1204: seq_num 753 ranges {528860=469053,1511801=517693,2512290=677867,3727346=1514} 2014-08-22 19:28:24,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1204: writing mira09725123-265 from 528860 to 997913 tid 1 2014-08-22 19:28:24,915.915 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:28:24,919.919 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.822881 7f4abe53c700 20 client.4227.objecter shutdown clearing up homeless session... 2014-08-22 19:28:24,919.919 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:17.823343 7f4abe53c700 1 librados: shutdown 2014-08-22 19:28:24,924.924 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:28:24,932.932 INFO:tasks.rados.rados.0.mira097.stdout:1204: writing mira09725123-265 from 1511801 to 2029494 tid 2 2014-08-22 19:28:25,064.064 INFO:tasks.rados.rados.0.mira097.stdout:1204: writing mira09725123-265 from 2512290 to 3190157 tid 3 2014-08-22 19:28:25,064.064 INFO:tasks.rados.rados.0.mira097.stdout:1204: writing mira09725123-265 from 3727346 to 3728860 tid 4 2014-08-22 19:28:25,064.064 INFO:tasks.rados.rados.0.mira097.stdout:1205: read oid 30 snap 60 2014-08-22 19:28:25,064.064 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 1 to mira09725123-265 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 2 to mira09725123-265 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1206: write oid 141 current snap is 67 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1206: seq_num 754 ranges {721241=440858,1521240=1} 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1206: writing mira09725123-141 from 721241 to 1162099 tid 1 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1206: writing mira09725123-141 from 1521240 to 1521241 tid 2 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1207: copy_from oid 274 from oid 488 current snap is 67 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1208: write oid 142 current snap is 67 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1208: seq_num 755 ranges {417495=564086,1217494=1} 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1208: writing mira09725123-142 from 417495 to 981581 tid 1 2014-08-22 19:28:25,065.065 INFO:tasks.rados.rados.0.mira097.stdout:1208: writing mira09725123-142 from 1217494 to 1217495 tid 2 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1209: read oid 4 snap -1 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 3 to mira09725123-265 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 4 to mira09725123-265 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 6 to mira09725123-265 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1204: finishing write tid 7 to mira09725123-265 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 265 v 1712 (ObjNum 753 snap 67 seq_num 753) dirty exists 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1204: done (6 left) 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1210: read oid 448 snap 60 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1211: copy_from oid 216 from oid 6 current snap is 67 2014-08-22 19:28:25,066.066 INFO:tasks.rados.rados.0.mira097.stdout:1212: delete oid 181 current snap is 67 2014-08-22 19:28:25,076.076 INFO:tasks.rados.rados.0.mira097.stdout:1203: expect (ObjNum 107 snap 0 seq_num 107) 2014-08-22 19:28:25,371.371 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.275302 7f6d278ec700 1 librados: starting msgr at :/0 2014-08-22 19:28:25,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.275307 7f6d278ec700 1 librados: starting objecter 2014-08-22 19:28:25,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.275438 7f6d278ec700 1 librados: setting wanted keys 2014-08-22 19:28:25,372.372 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.275445 7f6d278ec700 1 librados: calling monclient init 2014-08-22 19:28:25,373.373 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.277326 7f6d24999700 10 client.?.objecter ms_handle_connect 0x7f6d200355e0 2014-08-22 19:28:25,373.373 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.277351 7f6d24999700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:25,376.376 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.280535 7f6d278ec700 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:25,376.376 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.280647 7f6d278ec700 1 librados: init done 2014-08-22 19:28:25,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281407 7f6d24999700 10 client.4228.objecter ms_dispatch 0x7f6d20000f80 osd_map(191..191 src has 1..191) v3 2014-08-22 19:28:25,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281415 7f6d24999700 3 client.4228.objecter handle_osd_map got epochs [191,191] > 0 2014-08-22 19:28:25,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281417 7f6d24999700 3 client.4228.objecter handle_osd_map decoding full epoch 191 2014-08-22 19:28:25,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281500 7f6d24999700 20 client.4228.objecter dump_active .. 0 homeless 2014-08-22 19:28:25,378.378 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281619 7f6d24999700 10 client.4228.objecter ms_dispatch 0x7f6d20000f80 osd_map(191..191 src has 1..191) v3 2014-08-22 19:28:25,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281624 7f6d24999700 3 client.4228.objecter handle_osd_map ignoring epochs [191,191] <= 191 2014-08-22 19:28:25,379.379 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.281627 7f6d24999700 20 client.4228.objecter dump_active .. 0 homeless 2014-08-22 19:28:25,923.923 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:28:25,926.926 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.830465 7f6d278ec700 20 client.4228.objecter shutdown clearing up homeless session... 2014-08-22 19:28:25,926.926 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:18.830885 7f6d278ec700 1 librados: shutdown 2014-08-22 19:28:25,931.931 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:28:26,545.545 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.448540 7fd7f38d9700 1 librados: starting msgr at :/0 2014-08-22 19:28:26,545.545 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.448545 7fd7f38d9700 1 librados: starting objecter 2014-08-22 19:28:26,545.545 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.448688 7fd7f38d9700 1 librados: setting wanted keys 2014-08-22 19:28:26,545.545 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.448695 7fd7f38d9700 1 librados: calling monclient init 2014-08-22 19:28:26,547.547 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.450527 7fd7f0986700 10 client.?.objecter ms_handle_connect 0x7fd7ec0355e0 2014-08-22 19:28:26,547.547 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.450547 7fd7f0986700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:26,550.550 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.454101 7fd7f38d9700 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:26,550.550 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.454190 7fd7f38d9700 1 librados: init done 2014-08-22 19:28:26,551.551 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.454987 7fd7f0986700 10 client.4231.objecter ms_dispatch 0x7fd7ec000f80 osd_map(192..192 src has 1..192) v3 2014-08-22 19:28:26,648.648 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.454997 7fd7f0986700 3 client.4231.objecter handle_osd_map got epochs [192,192] > 0 2014-08-22 19:28:26,648.648 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.455001 7fd7f0986700 3 client.4231.objecter handle_osd_map decoding full epoch 192 2014-08-22 19:28:26,648.648 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.455095 7fd7f0986700 20 client.4231.objecter dump_active .. 0 homeless 2014-08-22 19:28:26,648.648 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.455137 7fd7f0986700 10 client.4231.objecter ms_dispatch 0x7fd7ec000f80 osd_map(192..192 src has 1..192) v3 2014-08-22 19:28:26,649.649 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.455142 7fd7f0986700 3 client.4231.objecter handle_osd_map ignoring epochs [192,192] <= 192 2014-08-22 19:28:26,649.649 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.455145 7fd7f0986700 20 client.4231.objecter dump_active .. 0 homeless 2014-08-22 19:28:26,757.757 INFO:tasks.rados.rados.0.mira097.stdout:1206: finishing write tid 1 to mira09725123-141 2014-08-22 19:28:26,757.757 INFO:tasks.rados.rados.0.mira097.stdout:1211: finishing copy_from to mira09725123-216 2014-08-22 19:28:26,757.757 INFO:tasks.rados.rados.0.mira097.stdout:1211: got expected ENOENT (src dne) 2014-08-22 19:28:26,757.757 INFO:tasks.rados.rados.0.mira097.stdout:1206: finishing write tid 2 to mira09725123-141 2014-08-22 19:28:26,758.758 INFO:tasks.rados.rados.0.mira097.stdout:1209: expect (ObjNum 3 snap 0 seq_num 3) 2014-08-22 19:28:26,944.944 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:28:26,947.947 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.851900 7fd7f38d9700 20 client.4231.objecter shutdown clearing up homeless session... 2014-08-22 19:28:26,947.947 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:19.852333 7fd7f38d9700 1 librados: shutdown 2014-08-22 19:28:26,952.952 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:28:27,338.338 INFO:tasks.rados.rados.0.mira097.stdout:1206: finishing write tid 4 to mira09725123-141 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1206: finishing write tid 5 to mira09725123-141 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 141 v 1784 (ObjNum 754 snap 67 seq_num 754) dirty exists 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1208: finishing write tid 1 to mira09725123-142 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1208: finishing write tid 2 to mira09725123-142 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1208: finishing write tid 4 to mira09725123-142 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1211: finishing copy_from racing read to mira09725123-216 2014-08-22 19:28:27,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1208: finishing write tid 5 to mira09725123-142 2014-08-22 19:28:27,340.340 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 142 v 1957 (ObjNum 755 snap 67 seq_num 755) dirty exists 2014-08-22 19:28:27,340.340 INFO:tasks.rados.rados.0.mira097.stdout:1210: expect (ObjNum 447 snap 0 seq_num 447) 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.470464 7fd06fd06700 1 librados: starting msgr at :/0 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.470470 7fd06fd06700 1 librados: starting objecter 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.470812 7fd06fd06700 1 librados: setting wanted keys 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.470823 7fd06fd06700 1 librados: calling monclient init 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.471622 7fd06cdb3700 10 client.?.objecter ms_handle_connect 0x7fd0680355e0 2014-08-22 19:28:27,567.567 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.471646 7fd06cdb3700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:27,568.568 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473131 7fd06fd06700 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:27,569.569 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473247 7fd06fd06700 1 librados: init done 2014-08-22 19:28:27,569.569 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473666 7fd06cdb3700 10 client.4230.objecter ms_dispatch 0x7fd068000f80 osd_map(193..193 src has 1..193) v3 2014-08-22 19:28:27,569.569 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473673 7fd06cdb3700 3 client.4230.objecter handle_osd_map got epochs [193,193] > 0 2014-08-22 19:28:27,569.569 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473674 7fd06cdb3700 3 client.4230.objecter handle_osd_map decoding full epoch 193 2014-08-22 19:28:27,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473739 7fd06cdb3700 20 client.4230.objecter dump_active .. 0 homeless 2014-08-22 19:28:27,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473764 7fd06cdb3700 10 client.4230.objecter ms_dispatch 0x7fd068000f80 osd_map(193..193 src has 1..193) v3 2014-08-22 19:28:27,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473768 7fd06cdb3700 3 client.4230.objecter handle_osd_map ignoring epochs [193,193] <= 193 2014-08-22 19:28:27,570.570 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.473770 7fd06cdb3700 20 client.4230.objecter dump_active .. 0 homeless 2014-08-22 19:28:27,961.961 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:28:27,964.964 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.868325 7fd06fd06700 20 client.4230.objecter shutdown clearing up homeless session... 2014-08-22 19:28:27,964.964 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:20.868776 7fd06fd06700 1 librados: shutdown 2014-08-22 19:28:27,969.969 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:28:28,268.268 INFO:tasks.rados.rados.0.mira097.stdout:1205: expect (ObjNum 29 snap 0 seq_num 29) 2014-08-22 19:28:28,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.486650 7f743bca3700 1 librados: starting msgr at :/0 2014-08-22 19:28:28,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.486657 7f743bca3700 1 librados: starting objecter 2014-08-22 19:28:28,583.583 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.486821 7f743bca3700 1 librados: setting wanted keys 2014-08-22 19:28:28,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.486825 7f743bca3700 1 librados: calling monclient init 2014-08-22 19:28:28,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.488607 7f7438d50700 10 client.?.objecter ms_handle_connect 0x7f74340355e0 2014-08-22 19:28:28,584.584 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.488638 7f7438d50700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:28,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.491657 7f743bca3700 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:28,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.491768 7f743bca3700 1 librados: init done 2014-08-22 19:28:28,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492630 7f7438d50700 10 client.4234.objecter ms_dispatch 0x7f7434000f80 osd_map(194..194 src has 1..194) v3 2014-08-22 19:28:28,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492639 7f7438d50700 3 client.4234.objecter handle_osd_map got epochs [194,194] > 0 2014-08-22 19:28:28,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492642 7f7438d50700 3 client.4234.objecter handle_osd_map decoding full epoch 194 2014-08-22 19:28:28,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492709 7f7438d50700 20 client.4234.objecter dump_active .. 0 homeless 2014-08-22 19:28:28,590.590 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492737 7f7438d50700 10 client.4234.objecter ms_dispatch 0x7f7434000f80 osd_map(194..194 src has 1..194) v3 2014-08-22 19:28:28,590.590 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492740 7f7438d50700 3 client.4234.objecter handle_osd_map ignoring epochs [194,194] <= 194 2014-08-22 19:28:28,590.590 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.492741 7f7438d50700 20 client.4234.objecter dump_active .. 0 homeless 2014-08-22 19:28:28,773.773 INFO:tasks.rados.rados.0.mira097.stdout:1207: finishing copy_from to mira09725123-274 2014-08-22 19:28:28,773.773 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 274 v 596 (ObjNum 487 snap 0 seq_num 487) dirty exists 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1207: finishing copy_from racing read to mira09725123-274 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1203: done (8 left) 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1205: done (7 left) 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1206: done (6 left) 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1207: done (5 left) 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1208: done (4 left) 2014-08-22 19:28:28,774.774 INFO:tasks.rados.rados.0.mira097.stdout:1209: done (3 left) 2014-08-22 19:28:28,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1210: done (2 left) 2014-08-22 19:28:28,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1211: done (1 left) 2014-08-22 19:28:28,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1212: done (0 left) 2014-08-22 19:28:28,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1213: write oid 335 current snap is 67 2014-08-22 19:28:28,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1213: seq_num 757 ranges {706131=543791,2028363=433546,2706130=1} 2014-08-22 19:28:28,799.799 INFO:tasks.rados.rados.0.mira097.stdout:1213: writing mira09725123-335 from 706131 to 1249922 tid 1 2014-08-22 19:28:28,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1213: writing mira09725123-335 from 2028363 to 2461909 tid 2 2014-08-22 19:28:28,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1213: writing mira09725123-335 from 2706130 to 2706131 tid 3 2014-08-22 19:28:28,819.819 INFO:tasks.rados.rados.0.mira097.stdout:1214: snap_create 2014-08-22 19:28:28,819.819 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:28,891.891 INFO:tasks.rados.rados.0.mira097.stdout:1213: finishing write tid 1 to mira09725123-335 2014-08-22 19:28:28,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1213: finishing write tid 2 to mira09725123-335 2014-08-22 19:28:28,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1213: finishing write tid 3 to mira09725123-335 2014-08-22 19:28:28,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1213: finishing write tid 5 to mira09725123-335 2014-08-22 19:28:28,893.893 INFO:tasks.rados.rados.0.mira097.stdout:1213: finishing write tid 6 to mira09725123-335 2014-08-22 19:28:28,893.893 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 335 v 1660 (ObjNum 757 snap 67 seq_num 757) dirty exists 2014-08-22 19:28:28,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1213: done (0 left) 2014-08-22 19:28:28,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1214: done (0 left) 2014-08-22 19:28:28,945.945 INFO:tasks.rados.rados.0.mira097.stdout:1215: delete oid 191 current snap is 68 2014-08-22 19:28:28,968.968 INFO:tasks.rados.rados.0.mira097.stdout:1215: done (0 left) 2014-08-22 19:28:28,968.968 INFO:tasks.rados.rados.0.mira097.stdout:1216: read oid 220 snap -1 2014-08-22 19:28:28,968.968 INFO:tasks.rados.rados.0.mira097.stdout:1217: read oid 483 snap -1 2014-08-22 19:28:28,969.969 INFO:tasks.rados.rados.0.mira097.stdout:1218: rollback oid 110 current snap is 68 2014-08-22 19:28:28,969.969 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 110 to 67 2014-08-22 19:28:28,969.969 INFO:tasks.rados.rados.0.mira097.stdout:1219: snap_create 2014-08-22 19:28:28,969.969 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:28:28,979.979 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 110 v 1671 (ObjNum 109 snap 0 seq_num 109) dirty exists 2014-08-22 19:28:28,980.980 INFO:tasks.rados.rados.0.mira097.stdout:1218: done (2 left) 2014-08-22 19:28:28,980.980 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:28,981.981 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:28:28,985.985 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.889106 7f743bca3700 20 client.4234.objecter shutdown clearing up homeless session... 2014-08-22 19:28:28,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:21.889739 7f743bca3700 1 librados: shutdown 2014-08-22 19:28:28,994.994 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:28:28,995.995 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:28:29,088.088 INFO:tasks.rados.rados.0.mira097.stdout:1216: expect (ObjNum 219 snap 0 seq_num 219) 2014-08-22 19:28:29,508.508 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.412199 7f9b196d1700 1 librados: starting msgr at :/0 2014-08-22 19:28:29,508.508 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.412204 7f9b196d1700 1 librados: starting objecter 2014-08-22 19:28:29,508.508 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.412632 7f9b196d1700 1 librados: setting wanted keys 2014-08-22 19:28:29,509.509 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.412639 7f9b196d1700 1 librados: calling monclient init 2014-08-22 19:28:29,509.509 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.413396 7f9b1258b700 10 client.?.objecter ms_handle_connect 0x7f9b140355e0 2014-08-22 19:28:29,509.509 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.413415 7f9b1258b700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:29,510.510 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.414729 7f9b196d1700 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:29,510.510 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.414834 7f9b196d1700 1 librados: init done 2014-08-22 19:28:29,510.510 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415071 7f9b1258b700 10 client.4217.objecter ms_dispatch 0x7f9b14000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:29,511.511 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415078 7f9b1258b700 3 client.4217.objecter handle_osd_map got epochs [195,195] > 0 2014-08-22 19:28:29,511.511 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415079 7f9b1258b700 3 client.4217.objecter handle_osd_map decoding full epoch 195 2014-08-22 19:28:29,511.511 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415145 7f9b1258b700 20 client.4217.objecter dump_active .. 0 homeless 2014-08-22 19:28:29,512.512 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415228 7f9b1258b700 10 client.4217.objecter ms_dispatch 0x7f9b14000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:29,512.512 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415233 7f9b1258b700 3 client.4217.objecter handle_osd_map ignoring epochs [195,195] <= 195 2014-08-22 19:28:29,512.512 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.415236 7f9b1258b700 20 client.4217.objecter dump_active .. 0 homeless 2014-08-22 19:28:29,595.595 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:28:30,039.039 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.782768 7f9b196d1700 20 client.4217.objecter shutdown clearing up homeless session... 2014-08-22 19:28:30,039.039 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:22.783607 7f9b196d1700 1 librados: shutdown 2014-08-22 19:28:30,125.125 INFO:tasks.rados.rados.0.mira097.stdout:1217: expect (ObjNum 531 snap 8 seq_num 531) 2014-08-22 19:28:30,222.222 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:28:30,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.673071 7f27b8464700 1 librados: starting msgr at :/0 2014-08-22 19:28:30,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.673078 7f27b8464700 1 librados: starting objecter 2014-08-22 19:28:30,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.673191 7f27b8464700 1 librados: setting wanted keys 2014-08-22 19:28:30,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.673196 7f27b8464700 1 librados: calling monclient init 2014-08-22 19:28:30,771.771 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.674102 7f27b5511700 10 client.?.objecter ms_handle_connect 0x7f27b00355e0 2014-08-22 19:28:30,771.771 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.674123 7f27b5511700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:30,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.675910 7f27b8464700 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:30,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676027 7f27b8464700 1 librados: init done 2014-08-22 19:28:30,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676371 7f27b5511700 10 client.4220.objecter ms_dispatch 0x7f27b0000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:30,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676381 7f27b5511700 3 client.4220.objecter handle_osd_map got epochs [195,195] > 0 2014-08-22 19:28:30,772.772 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676384 7f27b5511700 3 client.4220.objecter handle_osd_map decoding full epoch 195 2014-08-22 19:28:30,773.773 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676466 7f27b5511700 20 client.4220.objecter dump_active .. 0 homeless 2014-08-22 19:28:30,773.773 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676506 7f27b5511700 10 client.4220.objecter ms_dispatch 0x7f27b0000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:30,773.773 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676513 7f27b5511700 3 client.4220.objecter handle_osd_map ignoring epochs [195,195] <= 195 2014-08-22 19:28:30,774.774 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.676515 7f27b5511700 20 client.4220.objecter dump_active .. 0 homeless 2014-08-22 19:28:30,856.856 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:28:31,061.061 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.952282 7f27b8464700 20 client.4220.objecter shutdown clearing up homeless session... 2014-08-22 19:28:31,061.061 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:23.952800 7f27b8464700 1 librados: shutdown 2014-08-22 19:28:31,244.244 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:28:31,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1216: done (1 left) 2014-08-22 19:28:31,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1217: done (0 left) 2014-08-22 19:28:31,848.848 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.752113 7f119a7d9700 1 librados: starting msgr at :/0 2014-08-22 19:28:31,848.848 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.752121 7f119a7d9700 1 librados: starting objecter 2014-08-22 19:28:31,848.848 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.752393 7f119a7d9700 1 librados: setting wanted keys 2014-08-22 19:28:31,849.849 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.752412 7f119a7d9700 1 librados: calling monclient init 2014-08-22 19:28:31,850.850 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.753922 7f11937fe700 10 client.?.objecter ms_handle_connect 0x7f11940355e0 2014-08-22 19:28:31,850.850 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.753959 7f11937fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:31,853.853 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.756857 7f119a7d9700 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:31,853.853 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757038 7f119a7d9700 1 librados: init done 2014-08-22 19:28:31,854.854 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757560 7f11937fe700 10 client.4233.objecter ms_dispatch 0x7f1194000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:31,854.854 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757585 7f11937fe700 3 client.4233.objecter handle_osd_map got epochs [195,195] > 0 2014-08-22 19:28:31,854.854 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757592 7f11937fe700 3 client.4233.objecter handle_osd_map decoding full epoch 195 2014-08-22 19:28:31,854.854 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757734 7f11937fe700 20 client.4233.objecter dump_active .. 0 homeless 2014-08-22 19:28:31,855.855 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757785 7f11937fe700 10 client.4233.objecter ms_dispatch 0x7f1194000f80 osd_map(195..195 src has 1..195) v3 2014-08-22 19:28:31,855.855 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757790 7f11937fe700 3 client.4233.objecter handle_osd_map ignoring epochs [195,195] <= 195 2014-08-22 19:28:31,855.855 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.757793 7f11937fe700 20 client.4233.objecter dump_active .. 0 homeless 2014-08-22 19:28:31,937.937 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.841210 7f119a7d9700 20 client.4233.objecter shutdown clearing up homeless session... 2014-08-22 19:28:31,938.938 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:24.842051 7f119a7d9700 1 librados: shutdown 2014-08-22 19:28:31,946.946 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003310', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001678', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:27:35.668199', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873866268, u'health': u'HEALTH_OK', u'kb_used': 22914784, u'store_stats': {u'bytes_total': 10492975, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:27:36.469768', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 876956824, u'health': u'HEALTH_OK', u'kb_used': 19824228, u'store_stats': {u'bytes_total': 14687279, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}, {u'last_updated': u'2014-08-22 11:27:36.073290', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873866248, u'health': u'HEALTH_OK', u'kb_used': 22914804, u'store_stats': {u'bytes_total': 14687279, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65919, u'bytes_sst': 6832}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 84 > pgp_num 74'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.a has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 92, u'data_bytes': 1280548883, u'bytes_used': 3519250432, u'op_per_sec': 6, u'version': 314, u'pgs_by_state': [{u'count': 92, u'state_name': u'active+clean'}], u'read_bytes_sec': 506, u'write_bytes_sec': 484431, u'bytes_avail': 5994779795456}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 195, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:28:31,947.947 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:28:31,947.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:31,948.948 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.308686 2014-08-22 19:28:31,948.948 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.308685639814' 2014-08-22 19:28:32,314.314 INFO:tasks.rados.rados.0.mira097.stdout:1219: done (0 left) 2014-08-22 19:28:32,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1220: write oid 448 current snap is 69 2014-08-22 19:28:32,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1220: seq_num 759 ranges {462542=638823,1785458=725360,2960474=472466,3662541=1} 2014-08-22 19:28:32,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1220: writing mira09725123-448 from 462542 to 1101365 tid 1 2014-08-22 19:28:32,374.374 INFO:tasks.rados.rados.0.mira097.stdout:1220: writing mira09725123-448 from 1785458 to 2510818 tid 2 2014-08-22 19:28:32,390.390 INFO:tasks.rados.rados.0.mira097.stdout:1220: writing mira09725123-448 from 2960474 to 3432940 tid 3 2014-08-22 19:28:32,394.394 INFO:tasks.rados.rados.0.mira097.stdout:1220: writing mira09725123-448 from 3662541 to 3662542 tid 4 2014-08-22 19:28:32,395.395 INFO:tasks.rados.rados.0.mira097.stdout:1221: read oid 484 snap -1 2014-08-22 19:28:32,395.395 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 1 to mira09725123-448 2014-08-22 19:28:32,395.395 INFO:tasks.rados.rados.0.mira097.stdout:1222: snap_remove snap 67 2014-08-22 19:28:32,395.395 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 2 to mira09725123-448 2014-08-22 19:28:32,406.406 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 3 to mira09725123-448 2014-08-22 19:28:32,406.406 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 4 to mira09725123-448 2014-08-22 19:28:32,411.411 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 6 to mira09725123-448 2014-08-22 19:28:32,411.411 INFO:tasks.rados.rados.0.mira097.stdout:1220: finishing write tid 7 to mira09725123-448 2014-08-22 19:28:32,411.411 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 448 v 1725 (ObjNum 759 snap 69 seq_num 759) dirty exists 2014-08-22 19:28:32,411.411 INFO:tasks.rados.rados.0.mira097.stdout:1221: expect (ObjNum 483 snap 0 seq_num 483) 2014-08-22 19:28:32,625.625 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.529039 7f7dd0639700 1 librados: starting msgr at :/0 2014-08-22 19:28:32,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.529047 7f7dd0639700 1 librados: starting objecter 2014-08-22 19:28:32,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.529322 7f7dd0639700 1 librados: setting wanted keys 2014-08-22 19:28:32,626.626 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.529333 7f7dd0639700 1 librados: calling monclient init 2014-08-22 19:28:32,627.627 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.531620 7f7dcd6e6700 10 client.?.objecter ms_handle_connect 0x7f7dc80355e0 2014-08-22 19:28:32,627.627 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.531679 7f7dcd6e6700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:32,631.631 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.535409 7f7dd0639700 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:32,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.535595 7f7dd0639700 1 librados: init done 2014-08-22 19:28:32,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536364 7f7dcd6e6700 10 client.4237.objecter ms_dispatch 0x7f7dc8000f80 osd_map(196..196 src has 1..196) v3 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536378 7f7dcd6e6700 3 client.4237.objecter handle_osd_map got epochs [196,196] > 0 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536382 7f7dcd6e6700 3 client.4237.objecter handle_osd_map decoding full epoch 196 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536513 7f7dcd6e6700 20 client.4237.objecter dump_active .. 0 homeless 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536567 7f7dcd6e6700 10 client.4237.objecter ms_dispatch 0x7f7dc8000f80 osd_map(196..196 src has 1..196) v3 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536572 7f7dcd6e6700 3 client.4237.objecter handle_osd_map ignoring epochs [196,196] <= 196 2014-08-22 19:28:32,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:25.536575 7f7dcd6e6700 20 client.4237.objecter dump_active .. 0 homeless 2014-08-22 19:28:33,327.327 INFO:tasks.rados.rados.0.mira097.stdout:1220: done (2 left) 2014-08-22 19:28:33,327.327 INFO:tasks.rados.rados.0.mira097.stdout:1221: done (1 left) 2014-08-22 19:28:33,328.328 INFO:tasks.rados.rados.0.mira097.stdout:1222: done (0 left) 2014-08-22 19:28:33,328.328 INFO:tasks.rados.rados.0.mira097.stdout:1223: write oid 135 current snap is 69 2014-08-22 19:28:33,328.328 INFO:tasks.rados.rados.0.mira097.stdout:1223: seq_num 760 ranges {406347=1} 2014-08-22 19:28:33,334.334 INFO:tasks.rados.rados.0.mira097.stdout:1223: writing mira09725123-135 from 406347 to 406348 tid 1 2014-08-22 19:28:33,334.334 INFO:tasks.rados.rados.0.mira097.stdout:1224: write oid 189 current snap is 69 2014-08-22 19:28:33,334.334 INFO:tasks.rados.rados.0.mira097.stdout:1224: seq_num 761 ranges {102601=1} 2014-08-22 19:28:33,335.335 INFO:tasks.rados.rados.0.mira097.stdout:1224: writing mira09725123-189 from 102601 to 102602 tid 1 2014-08-22 19:28:33,335.335 INFO:tasks.rados.rados.0.mira097.stdout:1225: delete oid 258 current snap is 69 2014-08-22 19:28:33,336.336 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.308686 (8202302) 2014-08-22 19:28:33,341.341 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:26.245457 7f7dd0639700 20 client.4237.objecter shutdown clearing up homeless session... 2014-08-22 19:28:33,342.342 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:26.246035 7f7dd0639700 1 librados: shutdown 2014-08-22 19:28:33,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1224: finishing write tid 1 to mira09725123-189 2014-08-22 19:28:33,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1224: finishing write tid 3 to mira09725123-189 2014-08-22 19:28:33,365.365 INFO:tasks.rados.rados.0.mira097.stdout:1223: finishing write tid 1 to mira09725123-135 2014-08-22 19:28:33,366.366 INFO:tasks.rados.rados.0.mira097.stdout:1223: finishing write tid 3 to mira09725123-135 2014-08-22 19:28:33,366.366 INFO:tasks.rados.rados.0.mira097.stdout:1224: finishing write tid 4 to mira09725123-189 2014-08-22 19:28:33,366.366 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 189 v 1964 (ObjNum 761 snap 69 seq_num 761) dirty exists 2014-08-22 19:28:33,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1223: finishing write tid 4 to mira09725123-135 2014-08-22 19:28:33,373.373 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 135 v 1790 (ObjNum 760 snap 69 seq_num 760) dirty exists 2014-08-22 19:28:33,436.436 INFO:tasks.rados.rados.0.mira097.stdout:1223: done (2 left) 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1224: done (1 left) 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1225: done (0 left) 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1226: rollback oid 341 current snap is 69 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 341 to 60 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1227: copy_from oid 109 from oid 117 current snap is 69 2014-08-22 19:28:33,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1228: snap_remove snap 68 2014-08-22 19:28:33,482.482 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 341 v 1975 (ObjNum 293 snap 0 seq_num 293) dirty exists 2014-08-22 19:28:33,630.630 INFO:tasks.rados.rados.0.mira097.stdout:1227: finishing copy_from to mira09725123-109 2014-08-22 19:28:33,630.630 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 109 v 132 (ObjNum 116 snap 0 seq_num 116) dirty exists 2014-08-22 19:28:33,631.631 INFO:tasks.rados.rados.0.mira097.stdout:1227: finishing copy_from racing read to mira09725123-109 2014-08-22 19:28:34,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1226: done (2 left) 2014-08-22 19:28:34,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1227: done (1 left) 2014-08-22 19:28:34,339.339 INFO:tasks.rados.rados.0.mira097.stdout:1228: done (0 left) 2014-08-22 19:28:34,340.340 INFO:tasks.rados.rados.0.mira097.stdout:1229: read oid 10 snap -1 2014-08-22 19:28:34,340.340 INFO:tasks.rados.rados.0.mira097.stdout:1230: delete oid 381 current snap is 69 2014-08-22 19:28:34,360.360 INFO:tasks.rados.rados.0.mira097.stdout:1229: expect (ObjNum 9 snap 0 seq_num 9) 2014-08-22 19:28:34,380.380 INFO:tasks.rados.rados.0.mira097.stdout:1229: done (1 left) 2014-08-22 19:28:34,380.380 INFO:tasks.rados.rados.0.mira097.stdout:1230: done (0 left) 2014-08-22 19:28:34,380.380 INFO:tasks.rados.rados.0.mira097.stdout:1231: delete oid 75 current snap is 69 2014-08-22 19:28:34,404.404 INFO:tasks.rados.rados.0.mira097.stdout:1231: done (0 left) 2014-08-22 19:28:34,405.405 INFO:tasks.rados.rados.0.mira097.stdout:1232: rollback oid 308 current snap is 69 2014-08-22 19:28:34,405.405 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 308 to 60 2014-08-22 19:28:34,405.405 INFO:tasks.rados.rados.0.mira097.stdout:1233: rollback oid 409 current snap is 69 2014-08-22 19:28:34,405.405 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 409 to 60 2014-08-22 19:28:34,405.405 INFO:tasks.rados.rados.0.mira097.stdout:1234: rollback oid 354 current snap is 69 2014-08-22 19:28:34,406.406 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 354 to 60 2014-08-22 19:28:34,406.406 INFO:tasks.rados.rados.0.mira097.stdout:1235: delete oid 23 current snap is 69 2014-08-22 19:28:34,416.416 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 354 v 1751 (ObjNum 676 snap 51 seq_num 676) dirty exists 2014-08-22 19:28:34,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1234: done (3 left) 2014-08-22 19:28:34,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1235: done (2 left) 2014-08-22 19:28:34,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1236: read oid 9 snap -1 2014-08-22 19:28:34,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1237: rollback oid 379 current snap is 69 2014-08-22 19:28:34,432.432 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 379 to 60 2014-08-22 19:28:34,510.510 INFO:tasks.rados.rados.0.mira097.stdout:1238: snap_create 2014-08-22 19:28:34,510.510 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:28:34,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1236: expect (ObjNum 651 snap 43 seq_num 651) 2014-08-22 19:28:35,905.905 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 308 v 1754 (ObjNum 307 snap 0 seq_num 307) dirty exists 2014-08-22 19:28:35,906.906 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 379 v 1756 (ObjNum 732 snap 60 seq_num 732) dirty exists 2014-08-22 19:28:35,906.906 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 409 v 1680 (ObjNum 730 snap 60 seq_num 730) dirty exists 2014-08-22 19:28:35,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1232: done (3 left) 2014-08-22 19:28:35,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1233: done (2 left) 2014-08-22 19:28:35,906.906 INFO:tasks.rados.rados.0.mira097.stdout:1236: done (1 left) 2014-08-22 19:28:35,907.907 INFO:tasks.rados.rados.0.mira097.stdout:1237: done (0 left) 2014-08-22 19:28:36,361.361 INFO:tasks.rados.rados.0.mira097.stdout:1238: done (0 left) 2014-08-22 19:28:36,361.361 INFO:tasks.rados.rados.0.mira097.stdout:1239: read oid 104 snap -1 2014-08-22 19:28:36,361.361 INFO:tasks.rados.rados.0.mira097.stdout:1240: rollback oid 192 current snap is 70 2014-08-22 19:28:36,361.361 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 192 to 69 2014-08-22 19:28:36,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1241: copy_from oid 149 from oid 266 current snap is 70 2014-08-22 19:28:36,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1242: write oid 137 current snap is 70 2014-08-22 19:28:36,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1242: seq_num 766 ranges {643902=514670,1810367=433535} 2014-08-22 19:28:36,392.392 INFO:tasks.rados.rados.0.mira097.stdout:1242: writing mira09725123-137 from 643902 to 1158572 tid 1 2014-08-22 19:28:36,418.418 INFO:tasks.rados.rados.0.mira097.stdout:1242: writing mira09725123-137 from 1810367 to 2243902 tid 2 2014-08-22 19:28:36,418.418 INFO:tasks.rados.rados.0.mira097.stdout:1243: write oid 334 current snap is 70 2014-08-22 19:28:36,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1243: seq_num 767 ranges {704060=638922,1889427=424842,2918054=402502,3504059=1} 2014-08-22 19:28:36,442.442 INFO:tasks.rados.rados.0.mira097.stdout:1243: writing mira09725123-334 from 704060 to 1342982 tid 1 2014-08-22 19:28:36,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1243: writing mira09725123-334 from 1889427 to 2314269 tid 2 2014-08-22 19:28:36,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1243: writing mira09725123-334 from 2918054 to 3320556 tid 3 2014-08-22 19:28:36,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1243: writing mira09725123-334 from 3504059 to 3504060 tid 4 2014-08-22 19:28:36,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1244: write oid 69 current snap is 70 2014-08-22 19:28:36,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1244: seq_num 768 ranges {247865=1} 2014-08-22 19:28:36,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1244: writing mira09725123-69 from 247865 to 247866 tid 1 2014-08-22 19:28:36,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1245: snap_remove snap 60 2014-08-22 19:28:36,535.535 INFO:tasks.rados.rados.0.mira097.stdout:1239: expect (ObjNum 103 snap 0 seq_num 103) 2014-08-22 19:28:36,998.998 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 192 v 1997 (ObjNum 191 snap 0 seq_num 191) dirty exists 2014-08-22 19:28:36,998.998 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 1 to mira09725123-334 2014-08-22 19:28:36,998.998 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 2 to mira09725123-334 2014-08-22 19:28:36,998.998 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 3 to mira09725123-334 2014-08-22 19:28:36,998.998 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 4 to mira09725123-334 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 6 to mira09725123-334 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:1243: finishing write tid 7 to mira09725123-334 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 334 v 2007 (ObjNum 767 snap 70 seq_num 767) dirty exists 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:1242: finishing write tid 1 to mira09725123-137 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:1242: finishing write tid 2 to mira09725123-137 2014-08-22 19:28:36,999.999 INFO:tasks.rados.rados.0.mira097.stdout:1242: finishing write tid 4 to mira09725123-137 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:1242: finishing write tid 5 to mira09725123-137 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 137 v 1687 (ObjNum 766 snap 70 seq_num 766) dirty exists 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:1244: finishing write tid 1 to mira09725123-69 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:1244: finishing write tid 3 to mira09725123-69 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:1244: finishing write tid 4 to mira09725123-69 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 69 v 1690 (ObjNum 768 snap 70 seq_num 768) dirty exists 2014-08-22 19:28:37,000.000 INFO:tasks.rados.rados.0.mira097.stdout:1241: finishing copy_from to mira09725123-149 2014-08-22 19:28:37,001.001 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 149 v 1715 (ObjNum 265 snap 0 seq_num 265) dirty exists 2014-08-22 19:28:37,001.001 INFO:tasks.rados.rados.0.mira097.stdout:1241: finishing copy_from racing read to mira09725123-149 2014-08-22 19:28:37,372.372 INFO:tasks.rados.rados.0.mira097.stdout:1239: done (6 left) 2014-08-22 19:28:37,372.372 INFO:tasks.rados.rados.0.mira097.stdout:1240: done (5 left) 2014-08-22 19:28:37,372.372 INFO:tasks.rados.rados.0.mira097.stdout:1241: done (4 left) 2014-08-22 19:28:37,372.372 INFO:tasks.rados.rados.0.mira097.stdout:1242: done (3 left) 2014-08-22 19:28:37,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1243: done (2 left) 2014-08-22 19:28:37,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1244: done (1 left) 2014-08-22 19:28:37,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1245: done (0 left) 2014-08-22 19:28:37,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1246: snap_create 2014-08-22 19:28:38,348.348 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:38,349.349 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:38,349.349 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-22 19:28:38,349.349 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:28:38,349.349 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:28:38,350.350 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-22 19:28:38,378.378 INFO:tasks.rados.rados.0.mira097.stdout:1246: done (0 left) 2014-08-22 19:28:38,379.379 INFO:tasks.rados.rados.0.mira097.stdout:1247: delete oid 76 current snap is 71 2014-08-22 19:28:38,394.394 INFO:tasks.rados.rados.0.mira097.stdout:1247: done (0 left) 2014-08-22 19:28:38,394.394 INFO:tasks.rados.rados.0.mira097.stdout:1248: rollback oid 195 current snap is 71 2014-08-22 19:28:38,394.394 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 195 to 69 2014-08-22 19:28:38,394.394 INFO:tasks.rados.rados.0.mira097.stdout:1249: delete oid 51 current snap is 71 2014-08-22 19:28:38,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1249: done (1 left) 2014-08-22 19:28:38,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1250: snap_remove snap 70 2014-08-22 19:28:38,435.435 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 195 v 1716 (ObjNum 502 snap 2 seq_num 502) dirty exists 2014-08-22 19:28:39,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1248: done (1 left) 2014-08-22 19:28:39,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1250: done (0 left) 2014-08-22 19:28:39,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1251: read oid 211 snap 69 2014-08-22 19:28:39,400.400 INFO:tasks.rados.rados.0.mira097.stdout:1252: delete oid 192 current snap is 71 2014-08-22 19:28:39,421.421 INFO:tasks.rados.rados.0.mira097.stdout:1252: done (1 left) 2014-08-22 19:28:39,422.422 INFO:tasks.rados.rados.0.mira097.stdout:1253: rollback oid 315 current snap is 71 2014-08-22 19:28:39,422.422 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 315 to 69 2014-08-22 19:28:39,422.422 INFO:tasks.rados.rados.0.mira097.stdout:1254: write oid 121 current snap is 71 2014-08-22 19:28:39,422.422 INFO:tasks.rados.rados.0.mira097.stdout:1254: seq_num 772 ranges {444337=685464,1586873=57464} 2014-08-22 19:28:39,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1254: writing mira09725123-121 from 444337 to 1129801 tid 1 2014-08-22 19:28:39,457.457 INFO:tasks.rados.rados.0.mira097.stdout:1254: writing mira09725123-121 from 1586873 to 1644337 tid 2 2014-08-22 19:28:39,459.459 INFO:tasks.rados.rados.0.mira097.stdout:1255: read oid 321 snap 69 2014-08-22 19:28:39,460.460 INFO:tasks.rados.rados.0.mira097.stdout:1251: expect (ObjNum 503 snap 3 seq_num 503) 2014-08-22 19:28:40,361.361 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 315 v 2037 (ObjNum 679 snap 51 seq_num 679) dirty exists 2014-08-22 19:28:40,361.361 INFO:tasks.rados.rados.0.mira097.stdout:1255: expect (ObjNum 715 snap 54 seq_num 715) 2014-08-22 19:28:40,733.733 INFO:tasks.rados.rados.0.mira097.stdout:1254: finishing write tid 1 to mira09725123-121 2014-08-22 19:28:40,733.733 INFO:tasks.rados.rados.0.mira097.stdout:1254: finishing write tid 2 to mira09725123-121 2014-08-22 19:28:40,733.733 INFO:tasks.rados.rados.0.mira097.stdout:1254: finishing write tid 4 to mira09725123-121 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1254: finishing write tid 5 to mira09725123-121 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 121 v 1774 (ObjNum 772 snap 71 seq_num 772) dirty exists 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1251: done (3 left) 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1253: done (2 left) 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1254: done (1 left) 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1255: done (0 left) 2014-08-22 19:28:40,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1256: rollback oid 463 current snap is 71 2014-08-22 19:28:40,735.735 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 463 to 69 2014-08-22 19:28:40,735.735 INFO:tasks.rados.rados.0.mira097.stdout:1257: read oid 214 snap -1 2014-08-22 19:28:40,735.735 INFO:tasks.rados.rados.0.mira097.stdout:1258: delete oid 307 current snap is 71 2014-08-22 19:28:40,738.738 INFO:tasks.rados.rados.0.mira097.stdout:1257: expect (ObjNum 213 snap 0 seq_num 213) 2014-08-22 19:28:40,759.759 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 463 v 2040 (ObjNum 462 snap 0 seq_num 462) dirty exists 2014-08-22 19:28:40,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1256: done (2 left) 2014-08-22 19:28:40,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1257: done (1 left) 2014-08-22 19:28:40,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1258: done (0 left) 2014-08-22 19:28:40,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1259: delete oid 431 current snap is 71 2014-08-22 19:28:40,792.792 INFO:tasks.rados.rados.0.mira097.stdout:1259: done (0 left) 2014-08-22 19:28:40,793.793 INFO:tasks.rados.rados.0.mira097.stdout:1260: rollback oid 205 current snap is 71 2014-08-22 19:28:40,907.907 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 205 to 69 2014-08-22 19:28:40,907.907 INFO:tasks.rados.rados.0.mira097.stdout:1261: write oid 238 current snap is 71 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1261: seq_num 775 ranges {429226=636973,1626421=771875,2829225=1} 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1261: writing mira09725123-238 from 429226 to 1066199 tid 1 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1261: writing mira09725123-238 from 1626421 to 2398296 tid 2 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1261: writing mira09725123-238 from 2829225 to 2829226 tid 3 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:1262: snap_create 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:40,908.908 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 205 v 1779 (ObjNum 204 snap 0 seq_num 204) dirty exists 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1260: done (1 left) 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1261: finishing write tid 1 to mira09725123-238 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1261: finishing write tid 2 to mira09725123-238 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1261: finishing write tid 3 to mira09725123-238 2014-08-22 19:28:40,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1261: finishing write tid 5 to mira09725123-238 2014-08-22 19:28:41,052.052 INFO:tasks.rados.rados.0.mira097.stdout:1261: finishing write tid 6 to mira09725123-238 2014-08-22 19:28:41,053.053 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 238 v 2046 (ObjNum 775 snap 71 seq_num 775) dirty exists 2014-08-22 19:28:41,053.053 INFO:tasks.rados.rados.0.mira097.stdout:1261: done (0 left) 2014-08-22 19:28:41,449.449 INFO:tasks.rados.rados.0.mira097.stdout:1262: done (0 left) 2014-08-22 19:28:41,449.449 INFO:tasks.rados.rados.0.mira097.stdout:1263: read oid 442 snap -1 2014-08-22 19:28:41,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1264: delete oid 368 current snap is 72 2014-08-22 19:28:41,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1264: done (1 left) 2014-08-22 19:28:41,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1265: read oid 353 snap -1 2014-08-22 19:28:41,475.475 INFO:tasks.rados.rados.0.mira097.stdout:1266: read oid 104 snap 69 2014-08-22 19:28:41,478.478 INFO:tasks.rados.rados.0.mira097.stdout:1267: write oid 33 current snap is 72 2014-08-22 19:28:41,478.478 INFO:tasks.rados.rados.0.mira097.stdout:1267: seq_num 777 ranges {433189=400000} 2014-08-22 19:28:41,500.500 INFO:tasks.rados.rados.0.mira097.stdout:1267: writing mira09725123-33 from 433189 to 833189 tid 1 2014-08-22 19:28:41,500.500 INFO:tasks.rados.rados.0.mira097.stdout:1268: snap_remove snap 71 2014-08-22 19:28:41,548.548 INFO:tasks.rados.rados.0.mira097.stdout:1266: expect (ObjNum 103 snap 0 seq_num 103) 2014-08-22 19:28:42,032.032 INFO:tasks.rados.rados.0.mira097.stdout:1265: expect (ObjNum 352 snap 0 seq_num 352) 2014-08-22 19:28:42,378.378 INFO:tasks.rados.rados.0.mira097.stdout:1267: finishing write tid 1 to mira09725123-33 2014-08-22 19:28:42,378.378 INFO:tasks.rados.rados.0.mira097.stdout:1267: finishing write tid 3 to mira09725123-33 2014-08-22 19:28:42,378.378 INFO:tasks.rados.rados.0.mira097.stdout:1267: finishing write tid 4 to mira09725123-33 2014-08-22 19:28:42,379.379 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 33 v 1812 (ObjNum 777 snap 72 seq_num 777) dirty exists 2014-08-22 19:28:42,379.379 INFO:tasks.rados.rados.0.mira097.stdout:1263: expect (ObjNum 441 snap 0 seq_num 441) 2014-08-22 19:28:43,845.845 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:28:43,846.846 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:28:43,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1263: done (4 left) 2014-08-22 19:28:43,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1265: done (3 left) 2014-08-22 19:28:43,949.949 INFO:tasks.rados.rados.0.mira097.stdout:1266: done (2 left) 2014-08-22 19:28:43,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1267: done (1 left) 2014-08-22 19:28:43,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1268: done (0 left) 2014-08-22 19:28:43,950.950 INFO:tasks.rados.rados.0.mira097.stdout:1269: snap_remove snap 69 2014-08-22 19:28:44,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.222496 7f804096b700 1 librados: starting msgr at :/0 2014-08-22 19:28:44,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.222504 7f804096b700 1 librados: starting objecter 2014-08-22 19:28:44,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.223015 7f804096b700 1 librados: setting wanted keys 2014-08-22 19:28:44,320.320 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.223029 7f804096b700 1 librados: calling monclient init 2014-08-22 19:28:44,321.321 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.224458 7f80398ad700 10 client.?.objecter ms_handle_connect 0x7f803c0355e0 2014-08-22 19:28:44,321.321 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.224497 7f80398ad700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:44,323.323 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.227416 7f804096b700 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:44,324.324 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.227633 7f804096b700 1 librados: init done 2014-08-22 19:28:44,324.324 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228703 7f80398ad700 10 client.4236.objecter ms_dispatch 0x7f803c000f80 osd_map(204..204 src has 1..204) v3 2014-08-22 19:28:44,396.396 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228725 7f80398ad700 3 client.4236.objecter handle_osd_map got epochs [204,204] > 0 2014-08-22 19:28:44,396.396 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228731 7f80398ad700 3 client.4236.objecter handle_osd_map decoding full epoch 204 2014-08-22 19:28:44,396.396 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228887 7f80398ad700 20 client.4236.objecter dump_active .. 0 homeless 2014-08-22 19:28:44,397.397 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228953 7f80398ad700 10 client.4236.objecter ms_dispatch 0x7f803c000f80 osd_map(204..204 src has 1..204) v3 2014-08-22 19:28:44,397.397 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228961 7f80398ad700 3 client.4236.objecter handle_osd_map ignoring epochs [204,204] <= 204 2014-08-22 19:28:44,397.397 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.228963 7f80398ad700 20 client.4236.objecter dump_active .. 0 homeless 2014-08-22 19:28:44,469.469 INFO:tasks.rados.rados.0.mira097.stdout:1269: done (0 left) 2014-08-22 19:28:44,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1270: write oid 247 current snap is 72 2014-08-22 19:28:44,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1270: seq_num 778 ranges {529442=1} 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1270: writing mira09725123-247 from 529442 to 529443 tid 1 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1271: rollback oid 469 current snap is 72 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1271: done (1 left) 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1272: write oid 105 current snap is 72 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1272: seq_num 779 ranges {224672=1} 2014-08-22 19:28:44,495.495 INFO:tasks.rados.rados.0.mira097.stdout:1272: writing mira09725123-105 from 224672 to 224673 tid 1 2014-08-22 19:28:44,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1273: write oid 100 current snap is 72 2014-08-22 19:28:44,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1273: seq_num 780 ranges {534431=683515,1937370=597061} 2014-08-22 19:28:44,496.496 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:28:44,496.496 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.392756 7f804096b700 20 client.4236.objecter shutdown clearing up homeless session... 2014-08-22 19:28:44,496.496 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.393609 7f804096b700 1 librados: shutdown 2014-08-22 19:28:44,499.499 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 1 1' 2014-08-22 19:28:44,509.509 INFO:tasks.rados.rados.0.mira097.stdout:1273: writing mira09725123-100 from 534431 to 1217946 tid 1 2014-08-22 19:28:44,532.532 INFO:tasks.rados.rados.0.mira097.stdout:1273: writing mira09725123-100 from 1937370 to 2534431 tid 2 2014-08-22 19:28:44,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1274: copy_from oid 40 from oid 209 current snap is 72 2014-08-22 19:28:44,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1275: copy_from oid 217 from oid 186 current snap is 72 2014-08-22 19:28:44,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1276: snap_create 2014-08-22 19:28:44,537.537 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:28:44,543.543 INFO:tasks.rados.rados.0.mira097.stdout:1270: finishing write tid 1 to mira09725123-247 2014-08-22 19:28:44,546.546 INFO:tasks.rados.rados.0.mira097.stdout:1270: finishing write tid 3 to mira09725123-247 2014-08-22 19:28:44,554.554 INFO:tasks.rados.rados.0.mira097.stdout:1270: finishing write tid 4 to mira09725123-247 2014-08-22 19:28:44,555.555 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 247 v 1800 (ObjNum 778 snap 72 seq_num 778) dirty exists 2014-08-22 19:28:44,555.555 INFO:tasks.rados.rados.0.mira097.stdout:1270: done (4 left) 2014-08-22 19:28:44,556.556 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:28:44,572.572 INFO:tasks.rados.rados.0.mira097.stdout:1273: finishing write tid 1 to mira09725123-100 2014-08-22 19:28:44,573.573 INFO:tasks.rados.rados.0.mira097.stdout:1273: finishing write tid 2 to mira09725123-100 2014-08-22 19:28:44,573.573 INFO:tasks.rados.rados.0.mira097.stdout:1273: finishing write tid 4 to mira09725123-100 2014-08-22 19:28:44,602.602 INFO:tasks.rados.rados.0.mira097.stdout:1273: finishing write tid 5 to mira09725123-100 2014-08-22 19:28:44,602.602 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 100 v 2065 (ObjNum 780 snap 72 seq_num 780) dirty exists 2014-08-22 19:28:44,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1273: done (3 left) 2014-08-22 19:28:44,606.606 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:28:44,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1272: finishing write tid 1 to mira09725123-105 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1272: finishing write tid 3 to mira09725123-105 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1272: finishing write tid 4 to mira09725123-105 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 105 v 1803 (ObjNum 779 snap 72 seq_num 779) dirty exists 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1272: done (2 left) 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1275: finishing copy_from to mira09725123-217 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 217 v 540 (ObjNum 403 snap 0 seq_num 403) dirty exists 2014-08-22 19:28:44,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1275: finishing copy_from racing read to mira09725123-217 2014-08-22 19:28:44,644.644 INFO:tasks.rados.rados.0.mira097.stdout:1275: done (1 left) 2014-08-22 19:28:44,651.651 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:44,651.651 INFO:tasks.rados.rados.0.mira097.stdout:1274: finishing copy_from to mira09725123-40 2014-08-22 19:28:44,651.651 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 40 v 192 (ObjNum 208 snap 0 seq_num 208) dirty exists 2014-08-22 19:28:44,665.665 INFO:tasks.rados.rados.0.mira097.stdout:1274: finishing copy_from racing read to mira09725123-40 2014-08-22 19:28:44,665.665 INFO:tasks.rados.rados.0.mira097.stdout:1274: done (0 left) 2014-08-22 19:28:44,969.969 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.872165 7fc7a9b40700 1 librados: starting msgr at :/0 2014-08-22 19:28:44,969.969 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.872190 7fc7a9b40700 1 librados: starting objecter 2014-08-22 19:28:44,970.970 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.872527 7fc7a9b40700 1 librados: setting wanted keys 2014-08-22 19:28:44,970.970 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.872549 7fc7a9b40700 1 librados: calling monclient init 2014-08-22 19:28:44,970.970 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.874087 7fc7a27fc700 10 client.?.objecter ms_handle_connect 0x7fc7a40355e0 2014-08-22 19:28:44,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.874129 7fc7a27fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:44,973.973 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.876928 7fc7a9b40700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:44,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877114 7fc7a9b40700 1 librados: init done 2014-08-22 19:28:44,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877689 7fc7a27fc700 10 client.4239.objecter ms_dispatch 0x7fc7a4000f80 osd_map(205..205 src has 1..205) v3 2014-08-22 19:28:44,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877701 7fc7a27fc700 3 client.4239.objecter handle_osd_map got epochs [205,205] > 0 2014-08-22 19:28:44,974.974 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877704 7fc7a27fc700 3 client.4239.objecter handle_osd_map decoding full epoch 205 2014-08-22 19:28:44,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877882 7fc7a27fc700 20 client.4239.objecter dump_active .. 0 homeless 2014-08-22 19:28:44,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877955 7fc7a27fc700 10 client.4239.objecter ms_dispatch 0x7fc7a4000f80 osd_map(205..205 src has 1..205) v3 2014-08-22 19:28:44,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877964 7fc7a27fc700 3 client.4239.objecter handle_osd_map ignoring epochs [205,205] <= 205 2014-08-22 19:28:44,975.975 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:37.877966 7fc7a27fc700 20 client.4239.objecter dump_active .. 0 homeless 2014-08-22 19:28:45,479.479 INFO:tasks.rados.rados.0.mira097.stdout:1276: done (0 left) 2014-08-22 19:28:45,479.479 INFO:tasks.rados.rados.0.mira097.stdout:1277: snap_remove snap 72 2014-08-22 19:28:45,523.523 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.1 to 1 (8655362) 2014-08-22 19:28:45,527.527 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:38.431822 7fc7a9b40700 20 client.4239.objecter shutdown clearing up homeless session... 2014-08-22 19:28:45,528.528 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:38.432675 7fc7a9b40700 1 librados: shutdown 2014-08-22 19:28:45,538.538 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:28:46,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.079376 7fd753cf5700 1 librados: starting msgr at :/0 2014-08-22 19:28:46,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.079385 7fd753cf5700 1 librados: starting objecter 2014-08-22 19:28:46,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.079632 7fd753cf5700 1 librados: setting wanted keys 2014-08-22 19:28:46,177.177 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.079643 7fd753cf5700 1 librados: calling monclient init 2014-08-22 19:28:46,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.082213 7fd750da2700 10 client.?.objecter ms_handle_connect 0x7fd74c0355e0 2014-08-22 19:28:46,178.178 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.082310 7fd750da2700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:46,183.183 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.086320 7fd753cf5700 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:46,183.183 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.086450 7fd753cf5700 1 librados: init done 2014-08-22 19:28:46,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087392 7fd750da2700 10 client.4240.objecter ms_dispatch 0x7fd74c000f80 osd_map(206..206 src has 1..206) v3 2014-08-22 19:28:46,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087412 7fd750da2700 3 client.4240.objecter handle_osd_map got epochs [206,206] > 0 2014-08-22 19:28:46,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087419 7fd750da2700 3 client.4240.objecter handle_osd_map decoding full epoch 206 2014-08-22 19:28:46,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087579 7fd750da2700 20 client.4240.objecter dump_active .. 0 homeless 2014-08-22 19:28:46,254.254 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087666 7fd750da2700 10 client.4240.objecter ms_dispatch 0x7fd74c000f80 osd_map(206..206 src has 1..206) v3 2014-08-22 19:28:46,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087679 7fd750da2700 3 client.4240.objecter handle_osd_map ignoring epochs [206,206] <= 206 2014-08-22 19:28:46,255.255 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.087684 7fd750da2700 20 client.4240.objecter dump_active .. 0 homeless 2014-08-22 19:28:46,493.493 INFO:tasks.rados.rados.0.mira097.stdout:1277: done (0 left) 2014-08-22 19:28:46,494.494 INFO:tasks.rados.rados.0.mira097.stdout:1278: write oid 210 current snap is 73 2014-08-22 19:28:46,494.494 INFO:tasks.rados.rados.0.mira097.stdout:1278: seq_num 781 ranges {630684=406743,1519277=447318,2230683=1} 2014-08-22 19:28:46,519.519 INFO:tasks.rados.rados.0.mira097.stdout:1278: writing mira09725123-210 from 630684 to 1037427 tid 1 2014-08-22 19:28:46,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1278: writing mira09725123-210 from 1519277 to 1966595 tid 2 2014-08-22 19:28:46,536.536 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:28:46,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1278: writing mira09725123-210 from 2230683 to 2230684 tid 3 2014-08-22 19:28:46,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1279: read oid 123 snap -1 2014-08-22 19:28:46,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1280: rollback oid 255 current snap is 73 2014-08-22 19:28:46,538.538 INFO:tasks.rados.rados.0.mira097.stdout:1280: done (2 left) 2014-08-22 19:28:46,539.539 INFO:tasks.rados.rados.0.mira097.stdout:1281: write oid 446 current snap is 73 2014-08-22 19:28:46,539.539 INFO:tasks.rados.rados.0.mira097.stdout:1281: seq_num 782 ranges {725914=663219,1917057=8857} 2014-08-22 19:28:46,540.540 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.444071 7fd753cf5700 20 client.4240.objecter shutdown clearing up homeless session... 2014-08-22 19:28:46,540.540 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.444369 7fd753cf5700 1 librados: shutdown 2014-08-22 19:28:46,546.546 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:28:46,563.563 INFO:tasks.rados.rados.0.mira097.stdout:1281: writing mira09725123-446 from 725914 to 1389133 tid 1 2014-08-22 19:28:46,567.567 INFO:tasks.rados.rados.0.mira097.stdout:1281: writing mira09725123-446 from 1917057 to 1925914 tid 2 2014-08-22 19:28:46,567.567 INFO:tasks.rados.rados.0.mira097.stdout:1282: write oid 429 current snap is 73 2014-08-22 19:28:46,672.672 INFO:tasks.rados.rados.0.mira097.stdout:1282: seq_num 783 ranges {519320=635024,1576918=587709,2657469=547246,3719319=1} 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1282: writing mira09725123-429 from 519320 to 1154344 tid 1 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1282: writing mira09725123-429 from 1576918 to 2164627 tid 2 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1282: writing mira09725123-429 from 2657469 to 3204715 tid 3 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1282: writing mira09725123-429 from 3719319 to 3719320 tid 4 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1283: read oid 227 snap -1 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1278: finishing write tid 1 to mira09725123-210 2014-08-22 19:28:46,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1278: finishing write tid 2 to mira09725123-210 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1278: finishing write tid 3 to mira09725123-210 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1278: finishing write tid 5 to mira09725123-210 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1278: finishing write tid 6 to mira09725123-210 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 210 v 2072 (ObjNum 781 snap 73 seq_num 781) dirty exists 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1278: done (4 left) 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1284: write oid 12 current snap is 73 2014-08-22 19:28:46,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1284: seq_num 784 ranges {613526=606828,1805354=692976,3043142=370384} 2014-08-22 19:28:46,675.675 INFO:tasks.rados.rados.0.mira097.stdout:1284: writing mira09725123-12 from 613526 to 1220354 tid 1 2014-08-22 19:28:46,690.690 INFO:tasks.rados.rados.0.mira097.stdout:1284: writing mira09725123-12 from 1805354 to 2498330 tid 2 2014-08-22 19:28:46,705.705 INFO:tasks.rados.rados.0.mira097.stdout:1284: writing mira09725123-12 from 3043142 to 3413526 tid 3 2014-08-22 19:28:46,710.710 INFO:tasks.rados.rados.0.mira097.stdout:1285: rollback oid 220 current snap is 73 2014-08-22 19:28:46,710.710 INFO:tasks.rados.rados.0.mira097.stdout:1285: done (5 left) 2014-08-22 19:28:46,710.710 INFO:tasks.rados.rados.0.mira097.stdout:1286: write oid 346 current snap is 73 2014-08-22 19:28:46,710.710 INFO:tasks.rados.rados.0.mira097.stdout:1286: seq_num 785 ranges {426131=731080,1788285=763171,3114190=511941} 2014-08-22 19:28:46,732.732 INFO:tasks.rados.rados.0.mira097.stdout:1286: writing mira09725123-346 from 426131 to 1157211 tid 1 2014-08-22 19:28:46,761.761 INFO:tasks.rados.rados.0.mira097.stdout:1286: writing mira09725123-346 from 1788285 to 2551456 tid 2 2014-08-22 19:28:46,781.781 INFO:tasks.rados.rados.0.mira097.stdout:1286: writing mira09725123-346 from 3114190 to 3626131 tid 3 2014-08-22 19:28:46,782.782 INFO:tasks.rados.rados.0.mira097.stdout:1287: write oid 433 current snap is 73 2014-08-22 19:28:46,782.782 INFO:tasks.rados.rados.0.mira097.stdout:1287: seq_num 786 ranges {370960=1} 2014-08-22 19:28:46,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1287: writing mira09725123-433 from 370960 to 370961 tid 1 2014-08-22 19:28:46,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1288: read oid 157 snap -1 2014-08-22 19:28:46,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 1 to mira09725123-429 2014-08-22 19:28:46,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 2 to mira09725123-429 2014-08-22 19:28:46,783.783 INFO:tasks.rados.rados.0.mira097.stdout:1279: expect (ObjNum 122 snap 0 seq_num 122) 2014-08-22 19:28:47,005.005 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.907818 7fe49932f700 1 librados: starting msgr at :/0 2014-08-22 19:28:47,005.005 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.907824 7fe49932f700 1 librados: starting objecter 2014-08-22 19:28:47,005.005 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.907970 7fe49932f700 1 librados: setting wanted keys 2014-08-22 19:28:47,005.005 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.907975 7fe49932f700 1 librados: calling monclient init 2014-08-22 19:28:47,006.006 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.908712 7fe4922e8700 10 client.?.objecter ms_handle_connect 0x7fe4940355e0 2014-08-22 19:28:47,006.006 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.908739 7fe4922e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:47,006.006 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910321 7fe49932f700 10 client.4242.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:47,006.006 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910425 7fe49932f700 1 librados: init done 2014-08-22 19:28:47,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910739 7fe4922e8700 10 client.4242.objecter ms_dispatch 0x7fe494000f80 osd_map(207..207 src has 1..207) v3 2014-08-22 19:28:47,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910748 7fe4922e8700 3 client.4242.objecter handle_osd_map got epochs [207,207] > 0 2014-08-22 19:28:47,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910749 7fe4922e8700 3 client.4242.objecter handle_osd_map decoding full epoch 207 2014-08-22 19:28:47,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910824 7fe4922e8700 20 client.4242.objecter dump_active .. 0 homeless 2014-08-22 19:28:47,008.008 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910856 7fe4922e8700 10 client.4242.objecter ms_dispatch 0x7fe494000f80 osd_map(207..207 src has 1..207) v3 2014-08-22 19:28:47,008.008 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910858 7fe4922e8700 3 client.4242.objecter handle_osd_map ignoring epochs [207,207] <= 207 2014-08-22 19:28:47,008.008 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:39.910859 7fe4922e8700 20 client.4242.objecter dump_active .. 0 homeless 2014-08-22 19:28:47,548.548 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:28:47,550.550 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:40.454092 7fe49932f700 20 client.4242.objecter shutdown clearing up homeless session... 2014-08-22 19:28:47,550.550 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:40.454739 7fe49932f700 1 librados: shutdown 2014-08-22 19:28:47,557.557 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:28:48,172.172 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.075506 7f9f23c7c700 1 librados: starting msgr at :/0 2014-08-22 19:28:48,172.172 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.075514 7f9f23c7c700 1 librados: starting objecter 2014-08-22 19:28:48,173.173 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.075646 7f9f23c7c700 1 librados: setting wanted keys 2014-08-22 19:28:48,173.173 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.075653 7f9f23c7c700 1 librados: calling monclient init 2014-08-22 19:28:48,174.174 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.076373 7f9f20d29700 10 client.?.objecter ms_handle_connect 0x7f9f1c0355e0 2014-08-22 19:28:48,174.174 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.076395 7f9f20d29700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:48,174.174 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078135 7f9f23c7c700 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:48,174.174 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078235 7f9f23c7c700 1 librados: init done 2014-08-22 19:28:48,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078540 7f9f20d29700 10 client.4223.objecter ms_dispatch 0x7f9f1c000f80 osd_map(208..208 src has 1..208) v3 2014-08-22 19:28:48,175.175 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078548 7f9f20d29700 3 client.4223.objecter handle_osd_map got epochs [208,208] > 0 2014-08-22 19:28:48,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078550 7f9f20d29700 3 client.4223.objecter handle_osd_map decoding full epoch 208 2014-08-22 19:28:48,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078634 7f9f20d29700 20 client.4223.objecter dump_active .. 0 homeless 2014-08-22 19:28:48,176.176 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078676 7f9f20d29700 10 client.4223.objecter ms_dispatch 0x7f9f1c000f80 osd_map(208..208 src has 1..208) v3 2014-08-22 19:28:48,272.272 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078681 7f9f20d29700 3 client.4223.objecter handle_osd_map ignoring epochs [208,208] <= 208 2014-08-22 19:28:48,272.272 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.078683 7f9f20d29700 20 client.4223.objecter dump_active .. 0 homeless 2014-08-22 19:28:48,347.347 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 3 to mira09725123-429 2014-08-22 19:28:48,347.347 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 4 to mira09725123-429 2014-08-22 19:28:48,347.347 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 6 to mira09725123-429 2014-08-22 19:28:48,347.347 INFO:tasks.rados.rados.0.mira097.stdout:1282: finishing write tid 7 to mira09725123-429 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 429 v 1825 (ObjNum 783 snap 73 seq_num 783) dirty exists 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1284: finishing write tid 1 to mira09725123-12 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1281: finishing write tid 1 to mira09725123-446 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1281: finishing write tid 2 to mira09725123-446 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1281: finishing write tid 4 to mira09725123-446 2014-08-22 19:28:48,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1281: finishing write tid 5 to mira09725123-446 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 446 v 1724 (ObjNum 782 snap 73 seq_num 782) dirty exists 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:1284: finishing write tid 2 to mira09725123-12 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:1284: finishing write tid 3 to mira09725123-12 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:1284: finishing write tid 5 to mira09725123-12 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:1284: finishing write tid 6 to mira09725123-12 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 12 v 2082 (ObjNum 784 snap 73 seq_num 784) dirty exists 2014-08-22 19:28:48,349.349 INFO:tasks.rados.rados.0.mira097.stdout:1288: expect (ObjNum 156 snap 0 seq_num 156) 2014-08-22 19:28:48,615.615 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:28:48,619.619 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.522023 7f9f23c7c700 20 client.4223.objecter shutdown clearing up homeless session... 2014-08-22 19:28:48,619.619 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:41.522378 7f9f23c7c700 1 librados: shutdown 2014-08-22 19:28:48,623.623 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:28:49,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.141115 7f4174d0b700 1 librados: starting msgr at :/0 2014-08-22 19:28:49,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.141121 7f4174d0b700 1 librados: starting objecter 2014-08-22 19:28:49,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.141298 7f4174d0b700 1 librados: setting wanted keys 2014-08-22 19:28:49,238.238 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.141304 7f4174d0b700 1 librados: calling monclient init 2014-08-22 19:28:49,239.239 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.142103 7f416dd8a700 10 client.?.objecter ms_handle_connect 0x7f41700355e0 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.142123 7f416dd8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.143715 7f4174d0b700 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.143823 7f4174d0b700 1 librados: init done 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144297 7f416dd8a700 10 client.4245.objecter ms_dispatch 0x7f4170000f80 osd_map(209..209 src has 1..209) v3 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144307 7f416dd8a700 3 client.4245.objecter handle_osd_map got epochs [209,209] > 0 2014-08-22 19:28:49,422.422 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144310 7f416dd8a700 3 client.4245.objecter handle_osd_map decoding full epoch 209 2014-08-22 19:28:49,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144374 7f416dd8a700 20 client.4245.objecter dump_active .. 0 homeless 2014-08-22 19:28:49,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144403 7f416dd8a700 10 client.4245.objecter ms_dispatch 0x7f4170000f80 osd_map(209..209 src has 1..209) v3 2014-08-22 19:28:49,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144406 7f416dd8a700 3 client.4245.objecter handle_osd_map ignoring epochs [209,209] <= 209 2014-08-22 19:28:49,423.423 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.144408 7f416dd8a700 20 client.4245.objecter dump_active .. 0 homeless 2014-08-22 19:28:49,583.583 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:28:49,586.586 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.490162 7f4174d0b700 20 client.4245.objecter shutdown clearing up homeless session... 2014-08-22 19:28:49,586.586 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:42.490637 7f4174d0b700 1 librados: shutdown 2014-08-22 19:28:49,592.592 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:28:49,592.592 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:28:49,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1287: finishing write tid 1 to mira09725123-433 2014-08-22 19:28:49,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1287: finishing write tid 3 to mira09725123-433 2014-08-22 19:28:49,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1287: finishing write tid 4 to mira09725123-433 2014-08-22 19:28:49,918.918 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 433 v 2086 (ObjNum 786 snap 73 seq_num 786) dirty exists 2014-08-22 19:28:49,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1283: expect (ObjNum 226 snap 0 seq_num 226) 2014-08-22 19:28:50,207.207 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.110966 7f9cbc214700 1 librados: starting msgr at :/0 2014-08-22 19:28:50,208.208 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.110977 7f9cbc214700 1 librados: starting objecter 2014-08-22 19:28:50,208.208 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.111183 7f9cbc214700 1 librados: setting wanted keys 2014-08-22 19:28:50,208.208 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.111190 7f9cbc214700 1 librados: calling monclient init 2014-08-22 19:28:50,209.209 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.112902 7f9cb92c1700 10 client.?.objecter ms_handle_connect 0x7f9cb40355e0 2014-08-22 19:28:50,209.209 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.112927 7f9cb92c1700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:50,213.213 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.116733 7f9cbc214700 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:50,213.213 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.116832 7f9cbc214700 1 librados: init done 2014-08-22 19:28:50,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117726 7f9cb92c1700 10 client.4243.objecter ms_dispatch 0x7f9cb4000f80 osd_map(210..210 src has 1..210) v3 2014-08-22 19:28:50,214.214 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117737 7f9cb92c1700 3 client.4243.objecter handle_osd_map got epochs [210,210] > 0 2014-08-22 19:28:50,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117740 7f9cb92c1700 3 client.4243.objecter handle_osd_map decoding full epoch 210 2014-08-22 19:28:50,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117826 7f9cb92c1700 20 client.4243.objecter dump_active .. 0 homeless 2014-08-22 19:28:50,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117923 7f9cb92c1700 10 client.4243.objecter ms_dispatch 0x7f9cb4000f80 osd_map(210..210 src has 1..210) v3 2014-08-22 19:28:50,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117930 7f9cb92c1700 3 client.4243.objecter handle_osd_map ignoring epochs [210,210] <= 210 2014-08-22 19:28:50,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.117933 7f9cb92c1700 20 client.4243.objecter dump_active .. 0 homeless 2014-08-22 19:28:50,484.484 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:28:51,088.088 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.564356 7f9cbc214700 20 client.4243.objecter shutdown clearing up homeless session... 2014-08-22 19:28:51,088.088 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:43.564665 7f9cbc214700 1 librados: shutdown 2014-08-22 19:28:51,260.260 INFO:tasks.rados.rados.0.mira097.stdout:1286: finishing write tid 1 to mira09725123-346 2014-08-22 19:28:51,260.260 INFO:tasks.rados.rados.0.mira097.stdout:1286: finishing write tid 2 to mira09725123-346 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1286: finishing write tid 3 to mira09725123-346 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1286: finishing write tid 5 to mira09725123-346 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1286: finishing write tid 6 to mira09725123-346 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 346 v 1809 (ObjNum 785 snap 73 seq_num 785) dirty exists 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1279: done (7 left) 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1281: done (6 left) 2014-08-22 19:28:51,261.261 INFO:tasks.rados.rados.0.mira097.stdout:1282: done (5 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1283: done (4 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1284: done (3 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1286: done (2 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1287: done (1 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1288: done (0 left) 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1289: copy_from oid 426 from oid 33 current snap is 73 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1290: write oid 41 current snap is 73 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1290: seq_num 787 ranges {714767=400000} 2014-08-22 19:28:51,262.262 INFO:tasks.rados.rados.0.mira097.stdout:1290: writing mira09725123-41 from 714767 to 1114767 tid 1 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1291: read oid 380 snap -1 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1292: read oid 7 snap -1 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1293: delete oid 270 current snap is 73 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1289: finishing copy_from to mira09725123-426 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 426 v 1812 (ObjNum 777 snap 72 seq_num 777) dirty exists 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1289: finishing copy_from racing read to mira09725123-426 2014-08-22 19:28:51,263.263 INFO:tasks.rados.rados.0.mira097.stdout:1291: expect (ObjNum 213 snap 0 seq_num 213) 2014-08-22 19:28:51,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1289: done (4 left) 2014-08-22 19:28:51,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1291: done (3 left) 2014-08-22 19:28:51,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1293: done (2 left) 2014-08-22 19:28:51,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1294: snap_create 2014-08-22 19:28:51,433.433 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1290: finishing write tid 1 to mira09725123-41 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1290: finishing write tid 3 to mira09725123-41 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1290: finishing write tid 4 to mira09725123-41 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 41 v 1813 (ObjNum 787 snap 73 seq_num 787) dirty exists 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1290: done (1 left) 2014-08-22 19:28:51,434.434 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:28:51,598.598 INFO:tasks.rados.rados.0.mira097.stdout:1292: expect (ObjNum 6 snap 0 seq_num 6) 2014-08-22 19:28:51,611.611 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:28:52,262.262 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.164492 7fad12b81700 1 librados: starting msgr at :/0 2014-08-22 19:28:52,262.262 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.164496 7fad12b81700 1 librados: starting objecter 2014-08-22 19:28:52,262.262 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.164732 7fad12b81700 1 librados: setting wanted keys 2014-08-22 19:28:52,262.262 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.164740 7fad12b81700 1 librados: calling monclient init 2014-08-22 19:28:52,263.263 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.165695 7fad0b7fe700 10 client.?.objecter ms_handle_connect 0x7fad0c0355e0 2014-08-22 19:28:52,263.263 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.165716 7fad0b7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:52,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167197 7fad12b81700 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:52,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167296 7fad12b81700 1 librados: init done 2014-08-22 19:28:52,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167540 7fad0b7fe700 10 client.4248.objecter ms_dispatch 0x7fad0c000f80 osd_map(210..210 src has 1..210) v3 2014-08-22 19:28:52,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167546 7fad0b7fe700 3 client.4248.objecter handle_osd_map got epochs [210,210] > 0 2014-08-22 19:28:52,264.264 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167548 7fad0b7fe700 3 client.4248.objecter handle_osd_map decoding full epoch 210 2014-08-22 19:28:52,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167614 7fad0b7fe700 20 client.4248.objecter dump_active .. 0 homeless 2014-08-22 19:28:52,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167639 7fad0b7fe700 10 client.4248.objecter ms_dispatch 0x7fad0c000f80 osd_map(210..210 src has 1..210) v3 2014-08-22 19:28:52,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167641 7fad0b7fe700 3 client.4248.objecter handle_osd_map ignoring epochs [210,210] <= 210 2014-08-22 19:28:52,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.167643 7fad0b7fe700 20 client.4248.objecter dump_active .. 0 homeless 2014-08-22 19:28:52,446.446 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:28:53,047.047 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.525555 7fad12b81700 20 client.4248.objecter shutdown clearing up homeless session... 2014-08-22 19:28:53,047.047 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:45.525975 7fad12b81700 1 librados: shutdown 2014-08-22 19:28:53,318.318 INFO:tasks.rados.rados.0.mira097.stdout:1292: done (0 left) 2014-08-22 19:28:53,497.497 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:28:53,684.684 INFO:tasks.rados.rados.0.mira097.stdout:1294: done (0 left) 2014-08-22 19:28:53,684.684 INFO:tasks.rados.rados.0.mira097.stdout:1295: write oid 428 current snap is 74 2014-08-22 19:28:53,684.684 INFO:tasks.rados.rados.0.mira097.stdout:1295: seq_num 789 ranges {471178=405818,1469775=601403} 2014-08-22 19:28:53,706.706 INFO:tasks.rados.rados.0.mira097.stdout:1295: writing mira09725123-428 from 471178 to 876996 tid 1 2014-08-22 19:28:53,728.728 INFO:tasks.rados.rados.0.mira097.stdout:1295: writing mira09725123-428 from 1469775 to 2071178 tid 2 2014-08-22 19:28:53,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1296: write oid 356 current snap is 74 2014-08-22 19:28:53,732.732 INFO:tasks.rados.rados.0.mira097.stdout:1296: seq_num 790 ranges {416008=661270,1716131=465571,2695422=120586} 2014-08-22 19:28:53,751.751 INFO:tasks.rados.rados.0.mira097.stdout:1296: writing mira09725123-356 from 416008 to 1077278 tid 1 2014-08-22 19:28:53,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1296: writing mira09725123-356 from 1716131 to 2181702 tid 2 2014-08-22 19:28:53,775.775 INFO:tasks.rados.rados.0.mira097.stdout:1296: writing mira09725123-356 from 2695422 to 2816008 tid 3 2014-08-22 19:28:53,776.776 INFO:tasks.rados.rados.0.mira097.stdout:1297: read oid 444 snap 73 2014-08-22 19:28:53,776.776 INFO:tasks.rados.rados.0.mira097.stdout:1298: write oid 254 current snap is 74 2014-08-22 19:28:53,776.776 INFO:tasks.rados.rados.0.mira097.stdout:1298: seq_num 791 ranges {512262=785522,1699063=687190,2512261=1} 2014-08-22 19:28:53,803.803 INFO:tasks.rados.rados.0.mira097.stdout:1298: writing mira09725123-254 from 512262 to 1297784 tid 1 2014-08-22 19:28:53,825.825 INFO:tasks.rados.rados.0.mira097.stdout:1298: writing mira09725123-254 from 1699063 to 2386253 tid 2 2014-08-22 19:28:53,826.826 INFO:tasks.rados.rados.0.mira097.stdout:1298: writing mira09725123-254 from 2512261 to 2512262 tid 3 2014-08-22 19:28:53,828.828 INFO:tasks.rados.rados.0.mira097.stdout:1299: read oid 430 snap -1 2014-08-22 19:28:53,856.856 INFO:tasks.rados.rados.0.mira097.stdout:1295: finishing write tid 1 to mira09725123-428 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1295: finishing write tid 2 to mira09725123-428 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1295: finishing write tid 4 to mira09725123-428 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1296: finishing write tid 1 to mira09725123-356 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1296: finishing write tid 2 to mira09725123-356 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1296: finishing write tid 3 to mira09725123-356 2014-08-22 19:28:53,857.857 INFO:tasks.rados.rados.0.mira097.stdout:1296: finishing write tid 5 to mira09725123-356 2014-08-22 19:28:53,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1296: finishing write tid 6 to mira09725123-356 2014-08-22 19:28:53,858.858 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 356 v 1823 (ObjNum 790 snap 74 seq_num 790) dirty exists 2014-08-22 19:28:53,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1295: finishing write tid 5 to mira09725123-428 2014-08-22 19:28:53,858.858 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 428 v 2099 (ObjNum 789 snap 74 seq_num 789) dirty exists 2014-08-22 19:28:53,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1297: expect (ObjNum 443 snap 0 seq_num 443) 2014-08-22 19:28:53,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1298: finishing write tid 1 to mira09725123-254 2014-08-22 19:28:53,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1299: expect (ObjNum 490 snap 0 seq_num 490) 2014-08-22 19:28:53,895.895 INFO:tasks.rados.rados.0.mira097.stdout:1298: finishing write tid 2 to mira09725123-254 2014-08-22 19:28:53,895.895 INFO:tasks.rados.rados.0.mira097.stdout:1298: finishing write tid 3 to mira09725123-254 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1298: finishing write tid 5 to mira09725123-254 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1298: finishing write tid 6 to mira09725123-254 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 254 v 1735 (ObjNum 791 snap 74 seq_num 791) dirty exists 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1295: done (4 left) 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1296: done (3 left) 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1297: done (2 left) 2014-08-22 19:28:53,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1298: done (1 left) 2014-08-22 19:28:53,897.897 INFO:tasks.rados.rados.0.mira097.stdout:1299: done (0 left) 2014-08-22 19:28:53,897.897 INFO:tasks.rados.rados.0.mira097.stdout:1300: snap_create 2014-08-22 19:28:54,140.140 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.043370 7fc707316700 1 librados: starting msgr at :/0 2014-08-22 19:28:54,141.141 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.043394 7fc707316700 1 librados: starting objecter 2014-08-22 19:28:54,141.141 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.043711 7fc707316700 1 librados: setting wanted keys 2014-08-22 19:28:54,141.141 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.043724 7fc707316700 1 librados: calling monclient init 2014-08-22 19:28:54,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.045335 7fc6f7fff700 10 client.?.objecter ms_handle_connect 0x7fc7000355e0 2014-08-22 19:28:54,142.142 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.045384 7fc6f7fff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:54,146.146 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.048677 7fc707316700 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:54,146.146 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.048830 7fc707316700 1 librados: init done 2014-08-22 19:28:54,146.146 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049316 7fc6f7fff700 10 client.4226.objecter ms_dispatch 0x7fc700000f80 osd_map(211..211 src has 1..211) v3 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049334 7fc6f7fff700 3 client.4226.objecter handle_osd_map got epochs [211,211] > 0 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049340 7fc6f7fff700 3 client.4226.objecter handle_osd_map decoding full epoch 211 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049478 7fc6f7fff700 20 client.4226.objecter dump_active .. 0 homeless 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049554 7fc6f7fff700 10 client.4226.objecter ms_dispatch 0x7fc700000f80 osd_map(211..211 src has 1..211) v3 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049562 7fc6f7fff700 3 client.4226.objecter handle_osd_map ignoring epochs [211,211] <= 211 2014-08-22 19:28:54,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.049565 7fc6f7fff700 20 client.4226.objecter dump_active .. 0 homeless 2014-08-22 19:28:54,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.143771 7fc707316700 20 client.4226.objecter shutdown clearing up homeless session... 2014-08-22 19:28:54,313.313 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.144581 7fc707316700 1 librados: shutdown 2014-08-22 19:28:54,314.314 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003310', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001678', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:28:35.668417', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873836740, u'health': u'HEALTH_OK', u'kb_used': 22944312, u'store_stats': {u'bytes_total': 9366311, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316148}}, {u'last_updated': u'2014-08-22 11:28:36.470339', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 876908784, u'health': u'HEALTH_OK', u'kb_used': 19872268, u'store_stats': {u'bytes_total': 10414896, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316157}}, {u'last_updated': u'2014-08-22 11:28:36.073532', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873836736, u'health': u'HEALTH_OK', u'kb_used': 22944316, u'store_stats': {u'bytes_total': 10414896, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316157}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 84 > pgp_num 74'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.c has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 5998299045888, u'num_pgs': 92, u'data_bytes': 1238164694, u'bytes_used': 3430436864, u'op_per_sec': 0, u'version': 350, u'pgs_by_state': [{u'count': 92, u'state_name': u'active+clean'}], u'read_bytes_sec': 0, u'bytes_avail': 5994868609024}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 211, u'num_in_osds': 6}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:28:54,315.315 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:28:54,315.315 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:28:54,315.315 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:28:54,315.315 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:28:54,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1300: done (0 left) 2014-08-22 19:28:54,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1301: copy_from oid 91 from oid 298 current snap is 75 2014-08-22 19:28:54,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1302: write oid 255 current snap is 75 2014-08-22 19:28:54,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1302: seq_num 792 ranges {455043=758351,1909323=507785,2964624=290419} 2014-08-22 19:28:54,723.723 INFO:tasks.rados.rados.0.mira097.stdout:1302: writing mira09725123-255 from 455043 to 1213394 tid 1 2014-08-22 19:28:54,742.742 INFO:tasks.rados.rados.0.mira097.stdout:1302: writing mira09725123-255 from 1909323 to 2417108 tid 2 2014-08-22 19:28:54,761.761 INFO:tasks.rados.rados.0.mira097.stdout:1302: writing mira09725123-255 from 2964624 to 3255043 tid 3 2014-08-22 19:28:54,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1303: read oid 57 snap -1 2014-08-22 19:28:54,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1304: write oid 384 current snap is 75 2014-08-22 19:28:54,764.764 INFO:tasks.rados.rados.0.mira097.stdout:1304: seq_num 793 ranges {799873=481579,2022431=710205,3181722=782064,3999872=1} 2014-08-22 19:28:54,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1304: writing mira09725123-384 from 799873 to 1281452 tid 1 2014-08-22 19:28:54,808.808 INFO:tasks.rados.rados.0.mira097.stdout:1304: writing mira09725123-384 from 2022431 to 2732636 tid 2 2014-08-22 19:28:54,835.835 INFO:tasks.rados.rados.0.mira097.stdout:1304: writing mira09725123-384 from 3181722 to 3963786 tid 3 2014-08-22 19:28:54,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1304: writing mira09725123-384 from 3999872 to 3999873 tid 4 2014-08-22 19:28:54,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1305: copy_from oid 338 from oid 344 current snap is 75 2014-08-22 19:28:54,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1303: done (4 left) 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:1306: rollback oid 165 current snap is 75 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 165 to 74 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:1307: read oid 209 snap 74 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:1308: read oid 139 snap -1 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:1309: write oid 185 current snap is 75 2014-08-22 19:28:54,842.842 INFO:tasks.rados.rados.0.mira097.stdout:1309: seq_num 794 ranges {212478=1} 2014-08-22 19:28:54,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1309: writing mira09725123-185 from 212478 to 212479 tid 1 2014-08-22 19:28:54,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1310: read oid 39 snap -1 2014-08-22 19:28:54,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1311: write oid 419 current snap is 75 2014-08-22 19:28:54,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1311: seq_num 795 ranges {556285=400000} 2014-08-22 19:28:54,859.859 INFO:tasks.rados.rados.0.mira097.stdout:1311: writing mira09725123-419 from 556285 to 956285 tid 1 2014-08-22 19:28:54,862.862 INFO:tasks.rados.rados.0.mira097.stdout:1312: snap_create 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 10 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1302: finishing write tid 1 to mira09725123-255 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1302: finishing write tid 2 to mira09725123-255 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1302: finishing write tid 3 to mira09725123-255 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1302: finishing write tid 5 to mira09725123-255 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1302: finishing write tid 6 to mira09725123-255 2014-08-22 19:28:54,863.863 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 255 v 1742 (ObjNum 792 snap 75 seq_num 792) dirty exists 2014-08-22 19:28:54,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1302: done (9 left) 2014-08-22 19:28:54,864.864 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 9 2014-08-22 19:28:54,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 1 to mira09725123-384 2014-08-22 19:28:54,875.875 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 2 to mira09725123-384 2014-08-22 19:28:54,887.887 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 3 to mira09725123-384 2014-08-22 19:28:54,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 4 to mira09725123-384 2014-08-22 19:28:54,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 6 to mira09725123-384 2014-08-22 19:28:54,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1304: finishing write tid 7 to mira09725123-384 2014-08-22 19:28:54,890.890 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 384 v 1755 (ObjNum 793 snap 75 seq_num 793) dirty exists 2014-08-22 19:28:54,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1304: done (8 left) 2014-08-22 19:28:54,890.890 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 8 2014-08-22 19:28:54,918.918 INFO:tasks.rados.rados.0.mira097.stdout:1309: finishing write tid 1 to mira09725123-185 2014-08-22 19:28:54,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1309: finishing write tid 3 to mira09725123-185 2014-08-22 19:28:54,929.929 INFO:tasks.rados.rados.0.mira097.stdout:1309: finishing write tid 4 to mira09725123-185 2014-08-22 19:28:54,930.930 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 185 v 1828 (ObjNum 794 snap 75 seq_num 794) dirty exists 2014-08-22 19:28:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:1309: done (7 left) 2014-08-22 19:28:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 7 2014-08-22 19:28:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:1301: finishing copy_from to mira09725123-91 2014-08-22 19:28:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 91 v 400 (ObjNum 297 snap 0 seq_num 297) dirty exists 2014-08-22 19:28:54,937.937 INFO:tasks.rados.rados.0.mira097.stdout:1301: finishing copy_from racing read to mira09725123-91 2014-08-22 19:28:54,938.938 INFO:tasks.rados.rados.0.mira097.stdout:1301: done (6 left) 2014-08-22 19:28:54,938.938 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:28:54,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1308: expect (ObjNum 741 snap 63 seq_num 741) 2014-08-22 19:28:54,985.985 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.888147 7f5849cd7700 1 librados: starting msgr at :/0 2014-08-22 19:28:54,985.985 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.888157 7f5849cd7700 1 librados: starting objecter 2014-08-22 19:28:54,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.888349 7f5849cd7700 1 librados: setting wanted keys 2014-08-22 19:28:54,986.986 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.888353 7f5849cd7700 1 librados: calling monclient init 2014-08-22 19:28:55,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.909710 7f58427fc700 10 client.?.objecter ms_handle_connect 0x7f58440355e0 2014-08-22 19:28:55,007.007 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.909735 7f58427fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:55,029.029 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.932062 7f5849cd7700 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:55,029.029 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.932165 7f5849cd7700 1 librados: init done 2014-08-22 19:28:55,034.034 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937629 7f58427fc700 10 client.4246.objecter ms_dispatch 0x7f5844000f80 osd_map(212..212 src has 1..212) v3 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937641 7f58427fc700 3 client.4246.objecter handle_osd_map got epochs [212,212] > 0 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937644 7f58427fc700 3 client.4246.objecter handle_osd_map decoding full epoch 212 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937728 7f58427fc700 20 client.4246.objecter dump_active .. 0 homeless 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937758 7f58427fc700 10 client.4246.objecter ms_dispatch 0x7f5844000f80 osd_map(212..212 src has 1..212) v3 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937760 7f58427fc700 3 client.4246.objecter handle_osd_map ignoring epochs [212,212] <= 212 2014-08-22 19:28:55,035.035 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:47.937762 7f58427fc700 20 client.4246.objecter dump_active .. 0 homeless 2014-08-22 19:28:55,128.128 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:28:56,913.913 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 165 v 2102 (ObjNum 534 snap 8 seq_num 534) dirty exists 2014-08-22 19:28:56,914.914 INFO:tasks.rados.rados.0.mira097.stdout:1310: expect (ObjNum 38 snap 0 seq_num 38) 2014-08-22 19:28:56,922.922 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:49.106840 7f5849cd7700 20 client.4246.objecter shutdown clearing up homeless session... 2014-08-22 19:28:56,923.923 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:49.107234 7f5849cd7700 1 librados: shutdown 2014-08-22 19:28:57,243.243 INFO:tasks.rados.rados.0.mira097.stdout:1311: finishing write tid 1 to mira09725123-419 2014-08-22 19:28:57,243.243 INFO:tasks.rados.rados.0.mira097.stdout:1311: finishing write tid 3 to mira09725123-419 2014-08-22 19:28:57,243.243 INFO:tasks.rados.rados.0.mira097.stdout:1307: expect (ObjNum 208 snap 0 seq_num 208) 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1305: finishing copy_from to mira09725123-338 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1305: got expected ENOENT (src dne) 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1305: finishing copy_from racing read to mira09725123-338 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1311: finishing write tid 4 to mira09725123-419 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 419 v 1841 (ObjNum 795 snap 75 seq_num 795) dirty exists 2014-08-22 19:28:57,606.606 INFO:tasks.rados.rados.0.mira097.stdout:1305: done (5 left) 2014-08-22 19:28:57,607.607 INFO:tasks.rados.rados.0.mira097.stdout:1306: done (4 left) 2014-08-22 19:28:57,607.607 INFO:tasks.rados.rados.0.mira097.stdout:1307: done (3 left) 2014-08-22 19:28:57,607.607 INFO:tasks.rados.rados.0.mira097.stdout:1308: done (2 left) 2014-08-22 19:28:57,607.607 INFO:tasks.rados.rados.0.mira097.stdout:1310: done (1 left) 2014-08-22 19:28:57,607.607 INFO:tasks.rados.rados.0.mira097.stdout:1311: done (0 left) 2014-08-22 19:28:57,613.613 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 84' 2014-08-22 19:28:57,735.735 INFO:tasks.rados.rados.0.mira097.stdout:1312: done (0 left) 2014-08-22 19:28:57,736.736 INFO:tasks.rados.rados.0.mira097.stdout:1313: snap_remove snap 75 2014-08-22 19:28:58,195.195 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.097651 7fd0fc3c0700 1 librados: starting msgr at :/0 2014-08-22 19:28:58,195.195 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.097660 7fd0fc3c0700 1 librados: starting objecter 2014-08-22 19:28:58,195.195 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.097874 7fd0fc3c0700 1 librados: setting wanted keys 2014-08-22 19:28:58,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.097884 7fd0fc3c0700 1 librados: calling monclient init 2014-08-22 19:28:58,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.099392 7fd0f946d700 10 client.?.objecter ms_handle_connect 0x7fd0f40355e0 2014-08-22 19:28:58,196.196 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.099436 7fd0f946d700 10 client.?.objecter resend_mon_ops 2014-08-22 19:28:58,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.102411 7fd0fc3c0700 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:28:58,199.199 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.102562 7fd0fc3c0700 1 librados: init done 2014-08-22 19:28:58,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103064 7fd0f946d700 10 client.4251.objecter ms_dispatch 0x7fd0f4000f80 osd_map(213..213 src has 1..213) v3 2014-08-22 19:28:58,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103081 7fd0f946d700 3 client.4251.objecter handle_osd_map got epochs [213,213] > 0 2014-08-22 19:28:58,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103088 7fd0f946d700 3 client.4251.objecter handle_osd_map decoding full epoch 213 2014-08-22 19:28:58,200.200 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103266 7fd0f946d700 20 client.4251.objecter dump_active .. 0 homeless 2014-08-22 19:28:58,201.201 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103323 7fd0f946d700 10 client.4251.objecter ms_dispatch 0x7fd0f4000f80 osd_map(213..213 src has 1..213) v3 2014-08-22 19:28:58,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103328 7fd0f946d700 3 client.4251.objecter handle_osd_map ignoring epochs [213,213] <= 213 2014-08-22 19:28:58,265.265 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.103331 7fd0f946d700 20 client.4251.objecter dump_active .. 0 homeless 2014-08-22 19:28:58,754.754 INFO:tasks.rados.rados.0.mira097.stdout:1313: done (0 left) 2014-08-22 19:28:58,754.754 INFO:tasks.rados.rados.0.mira097.stdout:1314: write oid 361 current snap is 76 2014-08-22 19:28:58,755.755 INFO:tasks.rados.rados.0.mira097.stdout:1314: seq_num 796 ranges {501114=681664,1701113=1} 2014-08-22 19:28:58,766.766 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 84 2014-08-22 19:28:58,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.673453 7fd0fc3c0700 20 client.4251.objecter shutdown clearing up homeless session... 2014-08-22 19:28:58,770.770 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:51.673895 7fd0fc3c0700 1 librados: shutdown 2014-08-22 19:28:58,782.782 INFO:tasks.rados.rados.0.mira097.stdout:1314: writing mira09725123-361 from 501114 to 1182778 tid 1 2014-08-22 19:28:58,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1314: writing mira09725123-361 from 1701113 to 1701114 tid 2 2014-08-22 19:28:58,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1315: snap_remove snap 74 2014-08-22 19:28:58,832.832 INFO:tasks.rados.rados.0.mira097.stdout:1314: finishing write tid 1 to mira09725123-361 2014-08-22 19:28:58,832.832 INFO:tasks.rados.rados.0.mira097.stdout:1314: finishing write tid 2 to mira09725123-361 2014-08-22 19:28:58,832.832 INFO:tasks.rados.rados.0.mira097.stdout:1314: finishing write tid 4 to mira09725123-361 2014-08-22 19:28:58,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1314: finishing write tid 5 to mira09725123-361 2014-08-22 19:28:58,841.841 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 361 v 1837 (ObjNum 796 snap 76 seq_num 796) dirty exists 2014-08-22 19:28:59,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1314: done (1 left) 2014-08-22 19:28:59,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1315: done (0 left) 2014-08-22 19:28:59,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1316: read oid 166 snap -1 2014-08-22 19:28:59,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1317: delete oid 55 current snap is 76 2014-08-22 19:28:59,837.837 INFO:tasks.rados.rados.0.mira097.stdout:1317: done (1 left) 2014-08-22 19:28:59,837.837 INFO:tasks.rados.rados.0.mira097.stdout:1318: delete oid 419 current snap is 76 2014-08-22 19:28:59,843.843 INFO:tasks.rados.rados.0.mira097.stdout:1318: done (1 left) 2014-08-22 19:28:59,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1319: read oid 172 snap -1 2014-08-22 19:28:59,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1320: write oid 37 current snap is 76 2014-08-22 19:28:59,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1320: seq_num 799 ranges {753780=784597,2202008=151772} 2014-08-22 19:28:59,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1320: writing mira09725123-37 from 753780 to 1538377 tid 1 2014-08-22 19:28:59,897.897 INFO:tasks.rados.rados.0.mira097.stdout:1320: writing mira09725123-37 from 2202008 to 2353780 tid 2 2014-08-22 19:28:59,898.898 INFO:tasks.rados.rados.0.mira097.stdout:1321: read oid 245 snap 73 2014-08-22 19:28:59,898.898 INFO:tasks.rados.rados.0.mira097.stdout:1322: snap_create 2014-08-22 19:28:59,898.898 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:28:59,954.954 INFO:tasks.rados.rados.0.mira097.stdout:1316: expect (ObjNum 260 snap 0 seq_num 260) 2014-08-22 19:29:00,612.612 INFO:tasks.rados.rados.0.mira097.stdout:1319: expect (ObjNum 171 snap 0 seq_num 171) 2014-08-22 19:29:01,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1320: finishing write tid 1 to mira09725123-37 2014-08-22 19:29:01,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1320: finishing write tid 2 to mira09725123-37 2014-08-22 19:29:01,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1320: finishing write tid 4 to mira09725123-37 2014-08-22 19:29:01,819.819 INFO:tasks.rados.rados.0.mira097.stdout:1321: expect (ObjNum 244 snap 0 seq_num 244) 2014-08-22 19:29:03,050.050 INFO:tasks.rados.rados.0.mira097.stdout:1320: finishing write tid 5 to mira09725123-37 2014-08-22 19:29:03,051.051 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 37 v 1860 (ObjNum 799 snap 76 seq_num 799) dirty exists 2014-08-22 19:29:03,051.051 INFO:tasks.rados.rados.0.mira097.stdout:1316: done (3 left) 2014-08-22 19:29:03,051.051 INFO:tasks.rados.rados.0.mira097.stdout:1319: done (2 left) 2014-08-22 19:29:03,051.051 INFO:tasks.rados.rados.0.mira097.stdout:1320: done (1 left) 2014-08-22 19:29:03,051.051 INFO:tasks.rados.rados.0.mira097.stdout:1321: done (0 left) 2014-08-22 19:29:03,110.110 INFO:tasks.rados.rados.0.mira097.stdout:1322: done (0 left) 2014-08-22 19:29:03,110.110 INFO:tasks.rados.rados.0.mira097.stdout:1323: read oid 83 snap 73 2014-08-22 19:29:03,110.110 INFO:tasks.rados.rados.0.mira097.stdout:1324: rollback oid 87 current snap is 77 2014-08-22 19:29:03,110.110 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 87 to 76 2014-08-22 19:29:03,110.110 INFO:tasks.rados.rados.0.mira097.stdout:1325: snap_create 2014-08-22 19:29:03,111.111 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:29:03,156.156 INFO:tasks.rados.rados.0.mira097.stdout:1323: expect (ObjNum 82 snap 0 seq_num 82) 2014-08-22 19:29:03,336.336 INFO:tasks.rados.rados.0.mira097.stdout:1323: done (1 left) 2014-08-22 19:29:03,336.336 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:03,337.337 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 87 v 1772 (ObjNum 552 snap 13 seq_num 552) dirty exists 2014-08-22 19:29:03,337.337 INFO:tasks.rados.rados.0.mira097.stdout:1324: done (0 left) 2014-08-22 19:29:03,776.776 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:03,776.776 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:03,777.777 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:29:03,777.777 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:29:04,132.132 INFO:tasks.rados.rados.0.mira097.stdout:1325: done (0 left) 2014-08-22 19:29:04,133.133 INFO:tasks.rados.rados.0.mira097.stdout:1326: snap_remove snap 76 2014-08-22 19:29:04,251.251 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.153701 7f626a7af700 1 librados: starting msgr at :/0 2014-08-22 19:29:04,251.251 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.153710 7f626a7af700 1 librados: starting objecter 2014-08-22 19:29:04,251.251 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.153964 7f626a7af700 1 librados: setting wanted keys 2014-08-22 19:29:04,252.252 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.153974 7f626a7af700 1 librados: calling monclient init 2014-08-22 19:29:04,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.156274 7f625b7fe700 10 client.?.objecter ms_handle_connect 0x7f62640355e0 2014-08-22 19:29:04,253.253 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.156332 7f625b7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:04,257.257 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.160251 7f626a7af700 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:04,257.257 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.160424 7f626a7af700 1 librados: init done 2014-08-22 19:29:04,258.258 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161314 7f625b7fe700 10 client.4249.objecter ms_dispatch 0x7f6264000f80 osd_map(217..217 src has 1..217) v3 2014-08-22 19:29:04,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161328 7f625b7fe700 3 client.4249.objecter handle_osd_map got epochs [217,217] > 0 2014-08-22 19:29:04,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161332 7f625b7fe700 3 client.4249.objecter handle_osd_map decoding full epoch 217 2014-08-22 19:29:04,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161462 7f625b7fe700 20 client.4249.objecter dump_active .. 0 homeless 2014-08-22 19:29:04,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161564 7f625b7fe700 10 client.4249.objecter ms_dispatch 0x7f6264000f80 osd_map(217..217 src has 1..217) v3 2014-08-22 19:29:04,306.306 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161571 7f625b7fe700 3 client.4249.objecter handle_osd_map ignoring epochs [217,217] <= 217 2014-08-22 19:29:04,306.306 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.161574 7f625b7fe700 20 client.4249.objecter dump_active .. 0 homeless 2014-08-22 19:29:04,436.436 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:29:04,824.824 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.521283 7f626a7af700 20 client.4249.objecter shutdown clearing up homeless session... 2014-08-22 19:29:04,824.824 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:57.522121 7f626a7af700 1 librados: shutdown 2014-08-22 19:29:05,063.063 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 84' 2014-08-22 19:29:05,148.148 INFO:tasks.rados.rados.0.mira097.stdout:1326: done (0 left) 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1327: write oid 490 current snap is 78 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1327: seq_num 800 ranges {146160=1} 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1327: writing mira09725123-490 from 146160 to 146161 tid 1 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1328: rollback oid 99 current snap is 78 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 99 to 73 2014-08-22 19:29:05,149.149 INFO:tasks.rados.rados.0.mira097.stdout:1329: read oid 160 snap 73 2014-08-22 19:29:05,150.150 INFO:tasks.rados.rados.0.mira097.stdout:1330: snap_create 2014-08-22 19:29:05,150.150 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:29:05,172.172 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 99 v 1787 (ObjNum 98 snap 0 seq_num 98) dirty exists 2014-08-22 19:29:05,172.172 INFO:tasks.rados.rados.0.mira097.stdout:1328: done (2 left) 2014-08-22 19:29:05,172.172 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:29:05,211.211 INFO:tasks.rados.rados.0.mira097.stdout:1327: finishing write tid 1 to mira09725123-490 2014-08-22 19:29:05,211.211 INFO:tasks.rados.rados.0.mira097.stdout:1327: finishing write tid 3 to mira09725123-490 2014-08-22 19:29:05,212.212 INFO:tasks.rados.rados.0.mira097.stdout:1327: finishing write tid 4 to mira09725123-490 2014-08-22 19:29:05,212.212 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 490 v 2113 (ObjNum 800 snap 78 seq_num 800) dirty exists 2014-08-22 19:29:05,212.212 INFO:tasks.rados.rados.0.mira097.stdout:1327: done (1 left) 2014-08-22 19:29:05,213.213 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:05,216.216 INFO:tasks.rados.rados.0.mira097.stdout:1329: expect (ObjNum 159 snap 0 seq_num 159) 2014-08-22 19:29:05,585.585 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.487714 7f56e34ae700 1 librados: starting msgr at :/0 2014-08-22 19:29:05,585.585 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.487720 7f56e34ae700 1 librados: starting objecter 2014-08-22 19:29:05,585.585 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.487899 7f56e34ae700 1 librados: setting wanted keys 2014-08-22 19:29:05,586.586 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.487904 7f56e34ae700 1 librados: calling monclient init 2014-08-22 19:29:05,586.586 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.488652 7f56dbfff700 10 client.?.objecter ms_handle_connect 0x7f56dc0355e0 2014-08-22 19:29:05,586.586 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.488674 7f56dbfff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:05,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490229 7f56e34ae700 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:05,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490303 7f56e34ae700 1 librados: init done 2014-08-22 19:29:05,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490601 7f56dbfff700 10 client.4254.objecter ms_dispatch 0x7f56dc000f80 osd_map(218..218 src has 1..218) v3 2014-08-22 19:29:05,587.587 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490608 7f56dbfff700 3 client.4254.objecter handle_osd_map got epochs [218,218] > 0 2014-08-22 19:29:05,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490610 7f56dbfff700 3 client.4254.objecter handle_osd_map decoding full epoch 218 2014-08-22 19:29:05,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490676 7f56dbfff700 20 client.4254.objecter dump_active .. 0 homeless 2014-08-22 19:29:05,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490805 7f56dbfff700 10 client.4254.objecter ms_dispatch 0x7f56dc000f80 osd_map(218..218 src has 1..218) v3 2014-08-22 19:29:05,588.588 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490811 7f56dbfff700 3 client.4254.objecter handle_osd_map ignoring epochs [218,218] <= 218 2014-08-22 19:29:05,589.589 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:58.490813 7f56dbfff700 20 client.4254.objecter dump_active .. 0 homeless 2014-08-22 19:29:05,656.656 INFO:tasks.rados.rados.0.mira097.stdout:1329: done (0 left) 2014-08-22 19:29:06,165.165 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 84 2014-08-22 19:29:06,167.167 INFO:tasks.rados.rados.0.mira097.stdout:1330: done (0 left) 2014-08-22 19:29:06,167.167 INFO:tasks.rados.rados.0.mira097.stdout:1331: copy_from oid 147 from oid 295 current snap is 79 2014-08-22 19:29:06,167.167 INFO:tasks.rados.rados.0.mira097.stdout:1332: copy_from oid 204 from oid 199 current snap is 79 2014-08-22 19:29:06,168.168 INFO:tasks.rados.rados.0.mira097.stdout:1333: snap_remove snap 78 2014-08-22 19:29:06,168.168 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:59.072406 7f56e34ae700 20 client.4254.objecter shutdown clearing up homeless session... 2014-08-22 19:29:06,169.169 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:28:59.072897 7f56e34ae700 1 librados: shutdown 2014-08-22 19:29:06,176.176 INFO:tasks.rados.rados.0.mira097.stdout:1331: finishing copy_from to mira09725123-147 2014-08-22 19:29:06,176.176 INFO:tasks.rados.rados.0.mira097.stdout:1331: got expected ENOENT (src dne) 2014-08-22 19:29:06,176.176 INFO:tasks.rados.rados.0.mira097.stdout:1331: finishing copy_from racing read to mira09725123-147 2014-08-22 19:29:06,181.181 INFO:tasks.rados.rados.0.mira097.stdout:1332: finishing copy_from to mira09725123-204 2014-08-22 19:29:06,181.181 INFO:tasks.rados.rados.0.mira097.stdout:1332: got expected ENOENT (src dne) 2014-08-22 19:29:06,182.182 INFO:tasks.rados.rados.0.mira097.stdout:1332: finishing copy_from racing read to mira09725123-204 2014-08-22 19:29:07,177.177 INFO:tasks.rados.rados.0.mira097.stdout:1331: done (2 left) 2014-08-22 19:29:07,178.178 INFO:tasks.rados.rados.0.mira097.stdout:1332: done (1 left) 2014-08-22 19:29:07,178.178 INFO:tasks.rados.rados.0.mira097.stdout:1333: done (0 left) 2014-08-22 19:29:07,178.178 INFO:tasks.rados.rados.0.mira097.stdout:1334: snap_create 2014-08-22 19:29:08,189.189 INFO:tasks.rados.rados.0.mira097.stdout:1334: done (0 left) 2014-08-22 19:29:08,189.189 INFO:tasks.rados.rados.0.mira097.stdout:1335: snap_remove snap 79 2014-08-22 19:29:09,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1335: done (0 left) 2014-08-22 19:29:09,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1336: copy_from oid 75 from oid 76 current snap is 80 2014-08-22 19:29:09,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1337: write oid 375 current snap is 80 2014-08-22 19:29:09,198.198 INFO:tasks.rados.rados.0.mira097.stdout:1337: seq_num 801 ranges {641391=479630,1840704=525014,2816485=713124,3841390=1} 2014-08-22 19:29:09,221.221 INFO:tasks.rados.rados.0.mira097.stdout:1337: writing mira09725123-375 from 641391 to 1121021 tid 1 2014-08-22 19:29:09,241.241 INFO:tasks.rados.rados.0.mira097.stdout:1337: writing mira09725123-375 from 1840704 to 2365718 tid 2 2014-08-22 19:29:09,267.267 INFO:tasks.rados.rados.0.mira097.stdout:1337: writing mira09725123-375 from 2816485 to 3529609 tid 3 2014-08-22 19:29:09,272.272 INFO:tasks.rados.rados.0.mira097.stdout:1337: writing mira09725123-375 from 3841390 to 3841391 tid 4 2014-08-22 19:29:09,273.273 INFO:tasks.rados.rados.0.mira097.stdout:1338: delete oid 272 current snap is 80 2014-08-22 19:29:09,273.273 INFO:tasks.rados.rados.0.mira097.stdout:1336: finishing copy_from to mira09725123-75 2014-08-22 19:29:09,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1336: got expected ENOENT (src dne) 2014-08-22 19:29:09,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1336: finishing copy_from racing read to mira09725123-75 2014-08-22 19:29:09,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 1 to mira09725123-375 2014-08-22 19:29:09,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 2 to mira09725123-375 2014-08-22 19:29:09,297.297 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 3 to mira09725123-375 2014-08-22 19:29:09,369.369 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 4 to mira09725123-375 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 6 to mira09725123-375 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1337: finishing write tid 7 to mira09725123-375 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 375 v 1858 (ObjNum 801 snap 80 seq_num 801) dirty exists 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1336: done (2 left) 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1337: done (1 left) 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1338: done (0 left) 2014-08-22 19:29:09,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1339: read oid 150 snap -1 2014-08-22 19:29:09,371.371 INFO:tasks.rados.rados.0.mira097.stdout:1340: copy_from oid 348 from oid 182 current snap is 80 2014-08-22 19:29:09,371.371 INFO:tasks.rados.rados.0.mira097.stdout:1341: delete oid 223 current snap is 80 2014-08-22 19:29:09,374.374 INFO:tasks.rados.rados.0.mira097.stdout:1339: expect (ObjNum 149 snap 0 seq_num 149) 2014-08-22 19:29:09,409.409 INFO:tasks.rados.rados.0.mira097.stdout:1339: done (2 left) 2014-08-22 19:29:09,409.409 INFO:tasks.rados.rados.0.mira097.stdout:1341: done (1 left) 2014-08-22 19:29:09,410.410 INFO:tasks.rados.rados.0.mira097.stdout:1342: write oid 333 current snap is 80 2014-08-22 19:29:09,410.410 INFO:tasks.rados.rados.0.mira097.stdout:1342: seq_num 804 ranges {742632=431139,1542631=1} 2014-08-22 19:29:09,428.428 INFO:tasks.rados.rados.0.mira097.stdout:1342: writing mira09725123-333 from 742632 to 1173771 tid 1 2014-08-22 19:29:09,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1342: writing mira09725123-333 from 1542631 to 1542632 tid 2 2014-08-22 19:29:09,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1343: snap_remove snap 77 2014-08-22 19:29:09,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1340: finishing copy_from to mira09725123-348 2014-08-22 19:29:09,431.431 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 348 v 27 (ObjNum 22 snap 0 seq_num 22) dirty exists 2014-08-22 19:29:09,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1340: finishing copy_from racing read to mira09725123-348 2014-08-22 19:29:09,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1342: finishing write tid 1 to mira09725123-333 2014-08-22 19:29:09,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1342: finishing write tid 2 to mira09725123-333 2014-08-22 19:29:09,452.452 INFO:tasks.rados.rados.0.mira097.stdout:1342: finishing write tid 4 to mira09725123-333 2014-08-22 19:29:09,462.462 INFO:tasks.rados.rados.0.mira097.stdout:1342: finishing write tid 5 to mira09725123-333 2014-08-22 19:29:09,463.463 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 333 v 1869 (ObjNum 804 snap 80 seq_num 804) dirty exists 2014-08-22 19:29:10,226.226 INFO:tasks.rados.rados.0.mira097.stdout:1340: done (2 left) 2014-08-22 19:29:10,226.226 INFO:tasks.rados.rados.0.mira097.stdout:1342: done (1 left) 2014-08-22 19:29:10,226.226 INFO:tasks.rados.rados.0.mira097.stdout:1343: done (0 left) 2014-08-22 19:29:10,227.227 INFO:tasks.rados.rados.0.mira097.stdout:1344: read oid 104 snap -1 2014-08-22 19:29:10,227.227 INFO:tasks.rados.rados.0.mira097.stdout:1345: write oid 301 current snap is 80 2014-08-22 19:29:10,227.227 INFO:tasks.rados.rados.0.mira097.stdout:1345: seq_num 805 ranges {437862=402943,1237861=1} 2014-08-22 19:29:10,249.249 INFO:tasks.rados.rados.0.mira097.stdout:1345: writing mira09725123-301 from 437862 to 840805 tid 1 2014-08-22 19:29:10,253.253 INFO:tasks.rados.rados.0.mira097.stdout:1345: writing mira09725123-301 from 1237861 to 1237862 tid 2 2014-08-22 19:29:10,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1346: write oid 58 current snap is 80 2014-08-22 19:29:10,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1346: seq_num 806 ranges {631268=659420,1922022=511062,2987285=43983} 2014-08-22 19:29:10,276.276 INFO:tasks.rados.rados.0.mira097.stdout:1346: writing mira09725123-58 from 631268 to 1290688 tid 1 2014-08-22 19:29:10,299.299 INFO:tasks.rados.rados.0.mira097.stdout:1346: writing mira09725123-58 from 1922022 to 2433084 tid 2 2014-08-22 19:29:10,397.397 INFO:tasks.rados.rados.0.mira097.stdout:1346: writing mira09725123-58 from 2987285 to 3031268 tid 3 2014-08-22 19:29:10,397.397 INFO:tasks.rados.rados.0.mira097.stdout:1347: read oid 296 snap -1 2014-08-22 19:29:10,398.398 INFO:tasks.rados.rados.0.mira097.stdout:1344: expect (ObjNum 103 snap 0 seq_num 103) 2014-08-22 19:29:10,819.819 INFO:tasks.rados.rados.0.mira097.stdout:1345: finishing write tid 1 to mira09725123-301 2014-08-22 19:29:10,820.820 INFO:tasks.rados.rados.0.mira097.stdout:1345: finishing write tid 2 to mira09725123-301 2014-08-22 19:29:10,820.820 INFO:tasks.rados.rados.0.mira097.stdout:1345: finishing write tid 4 to mira09725123-301 2014-08-22 19:29:10,820.820 INFO:tasks.rados.rados.0.mira097.stdout:1345: finishing write tid 5 to mira09725123-301 2014-08-22 19:29:10,820.820 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 301 v 2133 (ObjNum 805 snap 80 seq_num 805) dirty exists 2014-08-22 19:29:10,821.821 INFO:tasks.rados.rados.0.mira097.stdout:1346: finishing write tid 1 to mira09725123-58 2014-08-22 19:29:10,821.821 INFO:tasks.rados.rados.0.mira097.stdout:1347: expect (ObjNum 295 snap 0 seq_num 295) 2014-08-22 19:29:10,822.822 INFO:tasks.rados.rados.0.mira097.stdout:1346: finishing write tid 2 to mira09725123-58 2014-08-22 19:29:10,822.822 INFO:tasks.rados.rados.0.mira097.stdout:1346: finishing write tid 3 to mira09725123-58 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1346: finishing write tid 5 to mira09725123-58 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1346: finishing write tid 6 to mira09725123-58 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 58 v 1878 (ObjNum 806 snap 80 seq_num 806) dirty exists 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1344: done (3 left) 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1345: done (2 left) 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1346: done (1 left) 2014-08-22 19:29:10,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1347: done (0 left) 2014-08-22 19:29:10,824.824 INFO:tasks.rados.rados.0.mira097.stdout:1348: write oid 219 current snap is 80 2014-08-22 19:29:10,824.824 INFO:tasks.rados.rados.0.mira097.stdout:1348: seq_num 807 ranges {442850=782648,1751482=732681,3055262=187588} 2014-08-22 19:29:10,854.854 INFO:tasks.rados.rados.0.mira097.stdout:1348: writing mira09725123-219 from 442850 to 1225498 tid 1 2014-08-22 19:29:10,879.879 INFO:tasks.rados.rados.0.mira097.stdout:1348: writing mira09725123-219 from 1751482 to 2484163 tid 2 2014-08-22 19:29:10,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1348: writing mira09725123-219 from 3055262 to 3242850 tid 3 2014-08-22 19:29:10,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1349: snap_create 2014-08-22 19:29:10,993.993 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:10,993.993 INFO:tasks.rados.rados.0.mira097.stdout:1348: finishing write tid 1 to mira09725123-219 2014-08-22 19:29:10,993.993 INFO:tasks.rados.rados.0.mira097.stdout:1348: finishing write tid 2 to mira09725123-219 2014-08-22 19:29:10,993.993 INFO:tasks.rados.rados.0.mira097.stdout:1348: finishing write tid 3 to mira09725123-219 2014-08-22 19:29:10,993.993 INFO:tasks.rados.rados.0.mira097.stdout:1348: finishing write tid 5 to mira09725123-219 2014-08-22 19:29:10,994.994 INFO:tasks.rados.rados.0.mira097.stdout:1348: finishing write tid 6 to mira09725123-219 2014-08-22 19:29:10,994.994 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 219 v 1813 (ObjNum 807 snap 80 seq_num 807) dirty exists 2014-08-22 19:29:10,994.994 INFO:tasks.rados.rados.0.mira097.stdout:1348: done (0 left) 2014-08-22 19:29:11,175.175 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:11,175.175 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:11,176.176 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.598170422483 2014-08-22 19:29:11,176.176 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 0.598170422483' 2014-08-22 19:29:11,236.236 INFO:tasks.rados.rados.0.mira097.stdout:1349: done (0 left) 2014-08-22 19:29:11,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1350: delete oid 205 current snap is 81 2014-08-22 19:29:11,295.295 INFO:tasks.rados.rados.0.mira097.stdout:1350: done (0 left) 2014-08-22 19:29:11,295.295 INFO:tasks.rados.rados.0.mira097.stdout:1351: snap_remove snap 73 2014-08-22 19:29:11,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.548281 7f20adc2a700 1 librados: starting msgr at :/0 2014-08-22 19:29:11,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.548287 7f20adc2a700 1 librados: starting objecter 2014-08-22 19:29:11,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.548546 7f20adc2a700 1 librados: setting wanted keys 2014-08-22 19:29:11,646.646 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.548561 7f20adc2a700 1 librados: calling monclient init 2014-08-22 19:29:11,647.647 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.550763 7f20a77fe700 10 client.?.objecter ms_handle_connect 0x7f20a0030c60 2014-08-22 19:29:11,648.648 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.550799 7f20a77fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:11,651.651 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.554717 7f20adc2a700 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:11,652.652 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.554873 7f20adc2a700 1 librados: init done 2014-08-22 19:29:11,652.652 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.555770 7f20a77fe700 10 client.4252.objecter ms_dispatch 0x7f20a8000f80 osd_map(224..224 src has 1..224) v3 2014-08-22 19:29:11,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.555787 7f20a77fe700 3 client.4252.objecter handle_osd_map got epochs [224,224] > 0 2014-08-22 19:29:11,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.555792 7f20a77fe700 3 client.4252.objecter handle_osd_map decoding full epoch 224 2014-08-22 19:29:11,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.555958 7f20a77fe700 20 client.4252.objecter dump_active .. 0 homeless 2014-08-22 19:29:11,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.556045 7f20a77fe700 10 client.4252.objecter ms_dispatch 0x7f20a8000f80 osd_map(224..224 src has 1..224) v3 2014-08-22 19:29:11,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.556058 7f20a77fe700 3 client.4252.objecter handle_osd_map ignoring epochs [224,224] <= 224 2014-08-22 19:29:11,654.654 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:04.556063 7f20a77fe700 20 client.4252.objecter dump_active .. 0 homeless 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1351: done (0 left) 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1352: copy_from oid 341 from oid 148 current snap is 81 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1353: read oid 137 snap -1 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1354: copy_from oid 423 from oid 144 current snap is 81 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1355: write oid 119 current snap is 81 2014-08-22 19:29:12,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1355: seq_num 809 ranges {731484=1} 2014-08-22 19:29:12,252.252 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 0.59817 (8392012) 2014-08-22 19:29:12,252.252 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:05.155289 7f20adc2a700 20 client.4252.objecter shutdown clearing up homeless session... 2014-08-22 19:29:12,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1355: writing mira09725123-119 from 731484 to 731485 tid 1 2014-08-22 19:29:12,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1356: snap_create 2014-08-22 19:29:12,257.257 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:29:12,268.268 INFO:tasks.rados.rados.0.mira097.stdout:1353: expect (ObjNum 766 snap 70 seq_num 766) 2014-08-22 19:29:12,268.268 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:05.172060 7f20adc2a700 1 librados: shutdown 2014-08-22 19:29:13,837.837 INFO:tasks.rados.rados.0.mira097.stdout:1355: finishing write tid 1 to mira09725123-119 2014-08-22 19:29:13,837.837 INFO:tasks.rados.rados.0.mira097.stdout:1355: finishing write tid 3 to mira09725123-119 2014-08-22 19:29:13,837.837 INFO:tasks.rados.rados.0.mira097.stdout:1355: finishing write tid 4 to mira09725123-119 2014-08-22 19:29:13,837.837 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 119 v 1866 (ObjNum 809 snap 81 seq_num 809) dirty exists 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1352: finishing copy_from to mira09725123-341 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 341 v 151 (ObjNum 147 snap 0 seq_num 147) dirty exists 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1352: finishing copy_from racing read to mira09725123-341 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1352: done (3 left) 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1353: done (2 left) 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1355: done (1 left) 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:13,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1354: finishing copy_from to mira09725123-423 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 423 v 245 (ObjNum 230 snap 0 seq_num 230) dirty exists 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1354: finishing copy_from racing read to mira09725123-423 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1354: done (0 left) 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1356: done (0 left) 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:1357: rollback oid 210 current snap is 82 2014-08-22 19:29:13,839.839 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 210 to 81 2014-08-22 19:29:13,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1358: snap_create 2014-08-22 19:29:13,840.840 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:13,840.840 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 210 v 2148 (ObjNum 781 snap 73 seq_num 781) dirty exists 2014-08-22 19:29:13,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1357: done (0 left) 2014-08-22 19:29:14,723.723 INFO:tasks.rados.rados.0.mira097.stdout:1358: done (0 left) 2014-08-22 19:29:14,723.723 INFO:tasks.rados.rados.0.mira097.stdout:1359: write oid 169 current snap is 83 2014-08-22 19:29:14,723.723 INFO:tasks.rados.rados.0.mira097.stdout:1359: seq_num 810 ranges {426714=1} 2014-08-22 19:29:14,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1359: writing mira09725123-169 from 426714 to 426715 tid 1 2014-08-22 19:29:14,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1360: copy_from oid 230 from oid 191 current snap is 83 2014-08-22 19:29:14,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1361: rollback oid 126 current snap is 83 2014-08-22 19:29:14,727.727 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 126 to 80 2014-08-22 19:29:14,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1362: delete oid 154 current snap is 83 2014-08-22 19:29:14,760.760 INFO:tasks.rados.rados.0.mira097.stdout:1360: finishing copy_from to mira09725123-230 2014-08-22 19:29:14,760.760 INFO:tasks.rados.rados.0.mira097.stdout:1360: got expected ENOENT (src dne) 2014-08-22 19:29:14,899.899 INFO:tasks.rados.rados.0.mira097.stdout:1359: finishing write tid 1 to mira09725123-169 2014-08-22 19:29:14,899.899 INFO:tasks.rados.rados.0.mira097.stdout:1360: finishing copy_from racing read to mira09725123-230 2014-08-22 19:29:14,900.900 INFO:tasks.rados.rados.0.mira097.stdout:1359: finishing write tid 3 to mira09725123-169 2014-08-22 19:29:14,900.900 INFO:tasks.rados.rados.0.mira097.stdout:1359: finishing write tid 4 to mira09725123-169 2014-08-22 19:29:14,900.900 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 169 v 2153 (ObjNum 810 snap 83 seq_num 810) dirty exists 2014-08-22 19:29:14,900.900 INFO:tasks.rados.rados.0.mira097.stdout:1359: done (3 left) 2014-08-22 19:29:14,900.900 INFO:tasks.rados.rados.0.mira097.stdout:1360: done (2 left) 2014-08-22 19:29:14,901.901 INFO:tasks.rados.rados.0.mira097.stdout:1362: done (1 left) 2014-08-22 19:29:14,901.901 INFO:tasks.rados.rados.0.mira097.stdout:1363: snap_remove snap 81 2014-08-22 19:29:14,901.901 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 126 v 1826 (ObjNum 161 snap 0 seq_num 161) dirty exists 2014-08-22 19:29:17,185.185 INFO:tasks.rados.rados.0.mira097.stdout:1361: done (1 left) 2014-08-22 19:29:17,186.186 INFO:tasks.rados.rados.0.mira097.stdout:1363: done (0 left) 2014-08-22 19:29:17,186.186 INFO:tasks.rados.rados.0.mira097.stdout:1364: read oid 324 snap -1 2014-08-22 19:29:17,186.186 INFO:tasks.rados.rados.0.mira097.stdout:1365: read oid 95 snap -1 2014-08-22 19:29:17,187.187 INFO:tasks.rados.rados.0.mira097.stdout:1366: copy_from oid 244 from oid 68 current snap is 83 2014-08-22 19:29:17,187.187 INFO:tasks.rados.rados.0.mira097.stdout:1367: snap_remove snap 80 2014-08-22 19:29:17,220.220 INFO:tasks.rados.rados.0.mira097.stdout:1365: expect (ObjNum 694 snap 52 seq_num 694) 2014-08-22 19:29:17,289.289 INFO:tasks.thrashosds.thrasher:in_osds: [3, 1, 2, 0, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:17,289.289 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:17,290.290 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [3, 1, 2, 0, 4, 5] 2014-08-22 19:29:17,290.290 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 1' 2014-08-22 19:29:17,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.640961 7f4067b2f700 1 librados: starting msgr at :/0 2014-08-22 19:29:17,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.640967 7f4067b2f700 1 librados: starting objecter 2014-08-22 19:29:17,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.641091 7f4067b2f700 1 librados: setting wanted keys 2014-08-22 19:29:17,738.738 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.641097 7f4067b2f700 1 librados: calling monclient init 2014-08-22 19:29:17,739.739 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.642924 7f4064bdc700 10 client.?.objecter ms_handle_connect 0x7f40600355e0 2014-08-22 19:29:17,740.740 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.642962 7f4064bdc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:17,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.646998 7f4067b2f700 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:17,743.743 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.647083 7f4067b2f700 1 librados: init done 2014-08-22 19:29:17,842.842 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.647959 7f4064bdc700 10 client.4255.objecter ms_dispatch 0x7f4060000f80 osd_map(228..228 src has 1..228) v3 2014-08-22 19:29:17,842.842 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.647973 7f4064bdc700 3 client.4255.objecter handle_osd_map got epochs [228,228] > 0 2014-08-22 19:29:17,842.842 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.647979 7f4064bdc700 3 client.4255.objecter handle_osd_map decoding full epoch 228 2014-08-22 19:29:17,842.842 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.648076 7f4064bdc700 20 client.4255.objecter dump_active .. 0 homeless 2014-08-22 19:29:17,843.843 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.648134 7f4064bdc700 10 client.4255.objecter ms_dispatch 0x7f4060000f80 osd_map(228..228 src has 1..228) v3 2014-08-22 19:29:17,843.843 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.648139 7f4064bdc700 3 client.4255.objecter handle_osd_map ignoring epochs [228,228] <= 228 2014-08-22 19:29:17,843.843 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:10.648140 7f4064bdc700 20 client.4255.objecter dump_active .. 0 homeless 2014-08-22 19:29:18,076.076 INFO:tasks.rados.rados.0.mira097.stdout:1364: expect (ObjNum 323 snap 0 seq_num 323) 2014-08-22 19:29:18,239.239 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.1. 2014-08-22 19:29:18,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:11.144909 7f4067b2f700 20 client.4255.objecter shutdown clearing up homeless session... 2014-08-22 19:29:18,242.242 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:11.145174 7f4067b2f700 1 librados: shutdown 2014-08-22 19:29:18,830.830 INFO:tasks.rados.rados.0.mira097.stdout:1366: finishing copy_from to mira09725123-244 2014-08-22 19:29:18,830.830 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 244 v 49 (ObjNum 67 snap 0 seq_num 67) dirty exists 2014-08-22 19:29:18,830.830 INFO:tasks.rados.rados.0.mira097.stdout:1366: finishing copy_from racing read to mira09725123-244 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1364: done (3 left) 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1365: done (2 left) 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1366: done (1 left) 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1367: done (0 left) 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1368: copy_from oid 449 from oid 327 current snap is 83 2014-08-22 19:29:18,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1369: snap_remove snap 82 2014-08-22 19:29:20,382.382 INFO:tasks.rados.rados.0.mira097.stdout:1369: done (1 left) 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1370: read oid 277 snap -1 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1371: delete oid 368 current snap is 83 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1371: done (2 left) 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1372: write oid 283 current snap is 83 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1372: seq_num 813 ranges {527956=553442,1812658=315298} 2014-08-22 19:29:20,383.383 INFO:tasks.rados.rados.0.mira097.stdout:1372: writing mira09725123-283 from 527956 to 1081398 tid 1 2014-08-22 19:29:20,384.384 INFO:tasks.rados.rados.0.mira097.stdout:1372: writing mira09725123-283 from 1812658 to 2127956 tid 2 2014-08-22 19:29:20,384.384 INFO:tasks.rados.rados.0.mira097.stdout:1373: read oid 455 snap -1 2014-08-22 19:29:20,384.384 INFO:tasks.rados.rados.0.mira097.stdout:1374: write oid 319 current snap is 83 2014-08-22 19:29:20,384.384 INFO:tasks.rados.rados.0.mira097.stdout:1374: seq_num 814 ranges {624210=408894,1526790=297420} 2014-08-22 19:29:20,384.384 INFO:tasks.rados.rados.0.mira097.stdout:1374: writing mira09725123-319 from 624210 to 1033104 tid 1 2014-08-22 19:29:20,386.386 INFO:tasks.rados.rados.0.mira097.stdout:1374: writing mira09725123-319 from 1526790 to 1824210 tid 2 2014-08-22 19:29:20,386.386 INFO:tasks.rados.rados.0.mira097.stdout:1372: finishing write tid 1 to mira09725123-283 2014-08-22 19:29:20,386.386 INFO:tasks.rados.rados.0.mira097.stdout:1375: write oid 32 current snap is 83 2014-08-22 19:29:20,386.386 INFO:tasks.rados.rados.0.mira097.stdout:1375: seq_num 815 ranges {568015=532123,1639898=431139,2528488=39527} 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1375: writing mira09725123-32 from 568015 to 1100138 tid 1 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1375: writing mira09725123-32 from 1639898 to 2071037 tid 2 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1375: writing mira09725123-32 from 2528488 to 2568015 tid 3 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1376: read oid 276 snap -1 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1372: finishing write tid 2 to mira09725123-283 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1372: finishing write tid 4 to mira09725123-283 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:1372: finishing write tid 5 to mira09725123-283 2014-08-22 19:29:20,387.387 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 283 v 1848 (ObjNum 813 snap 83 seq_num 813) dirty exists 2014-08-22 19:29:20,388.388 INFO:tasks.rados.rados.0.mira097.stdout:1372: done (6 left) 2014-08-22 19:29:20,388.388 INFO:tasks.rados.rados.0.mira097.stdout:1377: write oid 63 current snap is 83 2014-08-22 19:29:20,557.557 INFO:tasks.rados.rados.0.mira097.stdout:1377: seq_num 816 ranges {629197=752503,1932462=652758,3210993=618204} 2014-08-22 19:29:20,557.557 INFO:tasks.rados.rados.0.mira097.stdout:1377: writing mira09725123-63 from 629197 to 1381700 tid 1 2014-08-22 19:29:20,557.557 INFO:tasks.rados.rados.0.mira097.stdout:1377: writing mira09725123-63 from 1932462 to 2585220 tid 2 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1377: writing mira09725123-63 from 3210993 to 3829197 tid 3 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1378: read oid 433 snap -1 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1368: finishing copy_from to mira09725123-449 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 449 v 438 (ObjNum 326 snap 0 seq_num 326) dirty exists 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1368: finishing copy_from racing read to mira09725123-449 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1370: expect (ObjNum 276 snap 0 seq_num 276) 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1375: finishing write tid 1 to mira09725123-32 2014-08-22 19:29:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1373: expect (ObjNum 454 snap 0 seq_num 454) 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1375: finishing write tid 2 to mira09725123-32 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1375: finishing write tid 3 to mira09725123-32 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1375: finishing write tid 5 to mira09725123-32 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1375: finishing write tid 6 to mira09725123-32 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 32 v 1913 (ObjNum 815 snap 83 seq_num 815) dirty exists 2014-08-22 19:29:21,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1378: expect (ObjNum 786 snap 73 seq_num 786) 2014-08-22 19:29:21,040.040 INFO:tasks.rados.rados.0.mira097.stdout:1376: expect (ObjNum 708 snap 52 seq_num 708) 2014-08-22 19:29:22,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1377: finishing write tid 1 to mira09725123-63 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1377: finishing write tid 2 to mira09725123-63 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1377: finishing write tid 3 to mira09725123-63 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1377: finishing write tid 5 to mira09725123-63 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1377: finishing write tid 6 to mira09725123-63 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 63 v 1854 (ObjNum 816 snap 83 seq_num 816) dirty exists 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1374: finishing write tid 1 to mira09725123-319 2014-08-22 19:29:22,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1374: finishing write tid 2 to mira09725123-319 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1374: finishing write tid 4 to mira09725123-319 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1374: finishing write tid 5 to mira09725123-319 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 319 v 1874 (ObjNum 814 snap 83 seq_num 814) dirty exists 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1368: done (7 left) 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1370: done (6 left) 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1373: done (5 left) 2014-08-22 19:29:22,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1374: done (4 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1375: done (3 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1376: done (2 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1377: done (1 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1378: done (0 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1379: rollback oid 158 current snap is 83 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1379: done (0 left) 2014-08-22 19:29:22,258.258 INFO:tasks.rados.rados.0.mira097.stdout:1380: read oid 460 snap -1 2014-08-22 19:29:22,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1381: rollback oid 53 current snap is 83 2014-08-22 19:29:22,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1381: done (1 left) 2014-08-22 19:29:22,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1382: copy_from oid 70 from oid 366 current snap is 83 2014-08-22 19:29:22,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1383: copy_from oid 334 from oid 166 current snap is 83 2014-08-22 19:29:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1384: rollback oid 480 current snap is 83 2014-08-22 19:29:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1384: done (3 left) 2014-08-22 19:29:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1385: read oid 426 snap -1 2014-08-22 19:29:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1386: write oid 493 current snap is 83 2014-08-22 19:29:22,430.430 INFO:tasks.rados.rados.0.mira097.stdout:1386: seq_num 817 ranges {723403=476755,1795970=571529,3142394=381009} 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1386: writing mira09725123-493 from 723403 to 1200158 tid 1 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1386: writing mira09725123-493 from 1795970 to 2367499 tid 2 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1386: writing mira09725123-493 from 3142394 to 3523403 tid 3 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1387: read oid 283 snap -1 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1388: read oid 60 snap -1 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1389: read oid 428 snap -1 2014-08-22 19:29:22,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1390: snap_create 2014-08-22 19:29:22,432.432 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 8 2014-08-22 19:29:22,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1385: expect (ObjNum 777 snap 72 seq_num 777) 2014-08-22 19:29:22,513.513 INFO:tasks.rados.rados.0.mira097.stdout:1387: expect (ObjNum 813 snap 83 seq_num 813) 2014-08-22 19:29:23,083.083 INFO:tasks.rados.rados.0.mira097.stdout:1383: finishing copy_from to mira09725123-334 2014-08-22 19:29:23,084.084 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 334 v 337 (ObjNum 260 snap 0 seq_num 260) dirty exists 2014-08-22 19:29:23,084.084 INFO:tasks.rados.rados.0.mira097.stdout:1383: finishing copy_from racing read to mira09725123-334 2014-08-22 19:29:23,084.084 INFO:tasks.rados.rados.0.mira097.stdout:1389: expect (ObjNum 789 snap 74 seq_num 789) 2014-08-22 19:29:23,246.246 INFO:tasks.thrashosds.thrasher:in_osds: [3, 2, 0, 4, 5] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:23,246.246 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:23,247.247 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:29:23,247.247 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:29:23,695.695 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.597196 7f8965c3d700 1 librados: starting msgr at :/0 2014-08-22 19:29:23,695.695 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.597204 7f8965c3d700 1 librados: starting objecter 2014-08-22 19:29:23,695.695 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.597356 7f8965c3d700 1 librados: setting wanted keys 2014-08-22 19:29:23,695.695 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.597360 7f8965c3d700 1 librados: calling monclient init 2014-08-22 19:29:23,696.696 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.599049 7f895e7fc700 10 client.?.objecter ms_handle_connect 0x7f89600355e0 2014-08-22 19:29:23,696.696 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.599069 7f895e7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:23,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.601724 7f8965c3d700 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:23,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.601842 7f8965c3d700 1 librados: init done 2014-08-22 19:29:23,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602549 7f895e7fc700 10 client.4258.objecter ms_dispatch 0x7f8960000f80 osd_map(230..230 src has 1..230) v3 2014-08-22 19:29:23,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602559 7f895e7fc700 3 client.4258.objecter handle_osd_map got epochs [230,230] > 0 2014-08-22 19:29:23,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602562 7f895e7fc700 3 client.4258.objecter handle_osd_map decoding full epoch 230 2014-08-22 19:29:23,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602647 7f895e7fc700 20 client.4258.objecter dump_active .. 0 homeless 2014-08-22 19:29:23,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602677 7f895e7fc700 10 client.4258.objecter ms_dispatch 0x7f8960000f80 osd_map(230..230 src has 1..230) v3 2014-08-22 19:29:23,701.701 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602683 7f895e7fc700 3 client.4258.objecter handle_osd_map ignoring epochs [230,230] <= 230 2014-08-22 19:29:23,701.701 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.602685 7f895e7fc700 20 client.4258.objecter dump_active .. 0 homeless 2014-08-22 19:29:23,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1380: expect (ObjNum 724 snap 58 seq_num 724) 2014-08-22 19:29:23,882.882 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:29:24,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.963554 7f8965c3d700 20 client.4258.objecter shutdown clearing up homeless session... 2014-08-22 19:29:24,241.241 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:16.963927 7f8965c3d700 1 librados: shutdown 2014-08-22 19:29:24,509.509 INFO:tasks.rados.rados.0.mira097.stdout:1382: finishing copy_from to mira09725123-70 2014-08-22 19:29:24,509.509 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 70 v 1515 (ObjNum 365 snap 0 seq_num 365) dirty exists 2014-08-22 19:29:24,509.509 INFO:tasks.rados.rados.0.mira097.stdout:1382: finishing copy_from racing read to mira09725123-70 2014-08-22 19:29:24,510.510 INFO:tasks.rados.rados.0.mira097.stdout:1388: expect (ObjNum 59 snap 0 seq_num 59) 2014-08-22 19:29:24,571.571 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 84' 2014-08-22 19:29:24,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1386: finishing write tid 1 to mira09725123-493 2014-08-22 19:29:24,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1386: finishing write tid 2 to mira09725123-493 2014-08-22 19:29:24,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1386: finishing write tid 3 to mira09725123-493 2014-08-22 19:29:24,867.867 INFO:tasks.rados.rados.0.mira097.stdout:1386: finishing write tid 5 to mira09725123-493 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1386: finishing write tid 6 to mira09725123-493 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 493 v 1884 (ObjNum 817 snap 83 seq_num 817) dirty exists 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1380: done (7 left) 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1382: done (6 left) 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1383: done (5 left) 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1385: done (4 left) 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1386: done (3 left) 2014-08-22 19:29:24,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1387: done (2 left) 2014-08-22 19:29:24,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1388: done (1 left) 2014-08-22 19:29:24,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1389: done (0 left) 2014-08-22 19:29:25,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.088113 7fca7cbcf700 1 librados: starting msgr at :/0 2014-08-22 19:29:25,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.088119 7fca7cbcf700 1 librados: starting objecter 2014-08-22 19:29:25,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.088324 7fca7cbcf700 1 librados: setting wanted keys 2014-08-22 19:29:25,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.088336 7fca7cbcf700 1 librados: calling monclient init 2014-08-22 19:29:25,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.089977 7fca75ae7700 10 client.?.objecter ms_handle_connect 0x7fca780355e0 2014-08-22 19:29:25,188.188 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:18.090020 7fca75ae7700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:26,224.224 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.123565 7fca7cbcf700 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:26,228.228 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.123704 7fca7cbcf700 1 librados: init done 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124527 7fca75ae7700 10 client.4229.objecter ms_dispatch 0x7fca78000f80 osd_map(231..231 src has 1..231) v3 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124537 7fca75ae7700 3 client.4229.objecter handle_osd_map got epochs [231,231] > 0 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124540 7fca75ae7700 3 client.4229.objecter handle_osd_map decoding full epoch 231 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124674 7fca75ae7700 20 client.4229.objecter dump_active .. 0 homeless 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124720 7fca75ae7700 10 client.4229.objecter ms_dispatch 0x7fca78000f80 osd_map(231..231 src has 1..231) v3 2014-08-22 19:29:26,229.229 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124725 7fca75ae7700 3 client.4229.objecter handle_osd_map ignoring epochs [231,231] <= 231 2014-08-22 19:29:26,230.230 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:19.124728 7fca75ae7700 20 client.4229.objecter dump_active .. 0 homeless 2014-08-22 19:29:26,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1390: done (0 left) 2014-08-22 19:29:26,731.731 INFO:tasks.rados.rados.0.mira097.stdout:1391: copy_from oid 87 from oid 438 current snap is 84 2014-08-22 19:29:26,731.731 INFO:tasks.rados.rados.0.mira097.stdout:1392: snap_create 2014-08-22 19:29:26,731.731 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:27,076.076 INFO:tasks.rados.rados.0.mira097.stdout:1391: finishing copy_from to mira09725123-87 2014-08-22 19:29:27,076.076 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 87 v 670 (ObjNum 437 snap 0 seq_num 437) dirty exists 2014-08-22 19:29:27,077.077 INFO:tasks.rados.rados.0.mira097.stdout:1391: finishing copy_from racing read to mira09725123-87 2014-08-22 19:29:27,077.077 INFO:tasks.rados.rados.0.mira097.stdout:1391: done (0 left) 2014-08-22 19:29:27,942.942 INFO:tasks.rados.rados.0.mira097.stdout:1392: done (0 left) 2014-08-22 19:29:27,942.942 INFO:tasks.rados.rados.0.mira097.stdout:1393: snap_create 2014-08-22 19:29:27,992.992 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 84 2014-08-22 19:29:27,995.995 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:20.896320 7fca7cbcf700 20 client.4229.objecter shutdown clearing up homeless session... 2014-08-22 19:29:27,996.996 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:20.897093 7fca7cbcf700 1 librados: shutdown 2014-08-22 19:29:28,953.953 INFO:tasks.rados.rados.0.mira097.stdout:1393: done (0 left) 2014-08-22 19:29:28,953.953 INFO:tasks.rados.rados.0.mira097.stdout:1394: write oid 438 current snap is 86 2014-08-22 19:29:28,953.953 INFO:tasks.rados.rados.0.mira097.stdout:1394: seq_num 818 ranges {268232=1} 2014-08-22 19:29:28,957.957 INFO:tasks.rados.rados.0.mira097.stdout:1394: writing mira09725123-438 from 268232 to 268233 tid 1 2014-08-22 19:29:28,958.958 INFO:tasks.rados.rados.0.mira097.stdout:1395: snap_create 2014-08-22 19:29:28,958.958 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:28,974.974 INFO:tasks.rados.rados.0.mira097.stdout:1394: finishing write tid 1 to mira09725123-438 2014-08-22 19:29:28,974.974 INFO:tasks.rados.rados.0.mira097.stdout:1394: finishing write tid 3 to mira09725123-438 2014-08-22 19:29:28,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1394: finishing write tid 4 to mira09725123-438 2014-08-22 19:29:28,986.986 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 438 v 1927 (ObjNum 818 snap 86 seq_num 818) dirty exists 2014-08-22 19:29:28,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1394: done (0 left) 2014-08-22 19:29:29,974.974 INFO:tasks.rados.rados.0.mira097.stdout:1395: done (0 left) 2014-08-22 19:29:29,974.974 INFO:tasks.rados.rados.0.mira097.stdout:1396: read oid 468 snap -1 2014-08-22 19:29:29,975.975 INFO:tasks.rados.rados.0.mira097.stdout:1397: snap_create 2014-08-22 19:29:29,975.975 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:29,994.994 INFO:tasks.rados.rados.0.mira097.stdout:1396: expect (ObjNum 467 snap 0 seq_num 467) 2014-08-22 19:29:30,054.054 INFO:tasks.rados.rados.0.mira097.stdout:1396: done (0 left) 2014-08-22 19:29:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1397: done (0 left) 2014-08-22 19:29:30,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1398: write oid 54 current snap is 88 2014-08-22 19:29:30,989.989 INFO:tasks.rados.rados.0.mira097.stdout:1398: seq_num 819 ranges {613062=400000} 2014-08-22 19:29:31,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1398: writing mira09725123-54 from 613062 to 1013062 tid 1 2014-08-22 19:29:31,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1399: read oid 233 snap -1 2014-08-22 19:29:31,011.011 INFO:tasks.rados.rados.0.mira097.stdout:1400: read oid 285 snap 84 2014-08-22 19:29:31,012.012 INFO:tasks.rados.rados.0.mira097.stdout:1401: write oid 26 current snap is 88 2014-08-22 19:29:31,012.012 INFO:tasks.rados.rados.0.mira097.stdout:1401: seq_num 820 ranges {708291=1} 2014-08-22 19:29:31,017.017 INFO:tasks.rados.rados.0.mira097.stdout:1401: writing mira09725123-26 from 708291 to 708292 tid 1 2014-08-22 19:29:31,017.017 INFO:tasks.rados.rados.0.mira097.stdout:1402: read oid 283 snap -1 2014-08-22 19:29:31,017.017 INFO:tasks.rados.rados.0.mira097.stdout:1403: write oid 279 current snap is 88 2014-08-22 19:29:31,018.018 INFO:tasks.rados.rados.0.mira097.stdout:1403: seq_num 821 ranges {769474=552517,1969473=1} 2014-08-22 19:29:31,160.160 INFO:tasks.rados.rados.0.mira097.stdout:1403: writing mira09725123-279 from 769474 to 1321991 tid 1 2014-08-22 19:29:31,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1403: writing mira09725123-279 from 1969473 to 1969474 tid 2 2014-08-22 19:29:31,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1404: snap_create 2014-08-22 19:29:31,161.161 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 6 2014-08-22 19:29:31,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1402: expect (ObjNum 813 snap 83 seq_num 813) 2014-08-22 19:29:31,592.592 INFO:tasks.rados.rados.0.mira097.stdout:1398: finishing write tid 1 to mira09725123-54 2014-08-22 19:29:31,592.592 INFO:tasks.rados.rados.0.mira097.stdout:1398: finishing write tid 3 to mira09725123-54 2014-08-22 19:29:31,592.592 INFO:tasks.rados.rados.0.mira097.stdout:1398: finishing write tid 4 to mira09725123-54 2014-08-22 19:29:31,592.592 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 54 v 2177 (ObjNum 819 snap 88 seq_num 819) dirty exists 2014-08-22 19:29:31,593.593 INFO:tasks.rados.rados.0.mira097.stdout:1401: finishing write tid 1 to mira09725123-26 2014-08-22 19:29:31,593.593 INFO:tasks.rados.rados.0.mira097.stdout:1401: finishing write tid 3 to mira09725123-26 2014-08-22 19:29:31,593.593 INFO:tasks.rados.rados.0.mira097.stdout:1401: finishing write tid 4 to mira09725123-26 2014-08-22 19:29:31,593.593 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 26 v 1891 (ObjNum 820 snap 88 seq_num 820) dirty exists 2014-08-22 19:29:31,593.593 INFO:tasks.rados.rados.0.mira097.stdout:1399: expect (ObjNum 232 snap 0 seq_num 232) 2014-08-22 19:29:32,338.338 INFO:tasks.rados.rados.0.mira097.stdout:1400: expect (ObjNum 284 snap 0 seq_num 284) 2014-08-22 19:29:33,008.008 INFO:tasks.thrashosds.thrasher:in_osds: [3, 2, 0, 4, 5] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:33,009.009 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 3 1' 2014-08-22 19:29:33,151.151 INFO:tasks.rados.rados.0.mira097.stdout:1403: finishing write tid 1 to mira09725123-279 2014-08-22 19:29:33,151.151 INFO:tasks.rados.rados.0.mira097.stdout:1403: finishing write tid 2 to mira09725123-279 2014-08-22 19:29:33,151.151 INFO:tasks.rados.rados.0.mira097.stdout:1403: finishing write tid 4 to mira09725123-279 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1403: finishing write tid 5 to mira09725123-279 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 279 v 1939 (ObjNum 821 snap 88 seq_num 821) dirty exists 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1398: done (5 left) 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1399: done (4 left) 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1400: done (3 left) 2014-08-22 19:29:33,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1401: done (2 left) 2014-08-22 19:29:33,153.153 INFO:tasks.rados.rados.0.mira097.stdout:1402: done (1 left) 2014-08-22 19:29:33,153.153 INFO:tasks.rados.rados.0.mira097.stdout:1403: done (0 left) 2014-08-22 19:29:33,210.210 INFO:tasks.rados.rados.0.mira097.stdout:1404: done (0 left) 2014-08-22 19:29:33,210.210 INFO:tasks.rados.rados.0.mira097.stdout:1405: snap_create 2014-08-22 19:29:33,483.483 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.385166 7f5a3ce1b700 1 librados: starting msgr at :/0 2014-08-22 19:29:33,484.484 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.385185 7f5a3ce1b700 1 librados: starting objecter 2014-08-22 19:29:33,484.484 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.385442 7f5a3ce1b700 1 librados: setting wanted keys 2014-08-22 19:29:33,484.484 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.385461 7f5a3ce1b700 1 librados: calling monclient init 2014-08-22 19:29:33,485.485 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.386994 7f5a35d8a700 10 client.?.objecter ms_handle_connect 0x7f5a380355e0 2014-08-22 19:29:33,485.485 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.387039 7f5a35d8a700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:33,487.487 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.389557 7f5a3ce1b700 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:33,487.487 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.389727 7f5a3ce1b700 1 librados: init done 2014-08-22 19:29:33,488.488 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390167 7f5a35d8a700 10 client.4232.objecter ms_dispatch 0x7f5a38000f80 osd_map(236..236 src has 1..236) v3 2014-08-22 19:29:33,488.488 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390176 7f5a35d8a700 3 client.4232.objecter handle_osd_map got epochs [236,236] > 0 2014-08-22 19:29:33,488.488 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390178 7f5a35d8a700 3 client.4232.objecter handle_osd_map decoding full epoch 236 2014-08-22 19:29:33,489.489 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390281 7f5a35d8a700 20 client.4232.objecter dump_active .. 0 homeless 2014-08-22 19:29:33,559.559 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390377 7f5a35d8a700 10 client.4232.objecter ms_dispatch 0x7f5a38000f80 osd_map(236..236 src has 1..236) v3 2014-08-22 19:29:33,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390383 7f5a35d8a700 3 client.4232.objecter handle_osd_map ignoring epochs [236,236] <= 236 2014-08-22 19:29:33,560.560 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:26.390386 7f5a35d8a700 20 client.4232.objecter dump_active .. 0 homeless 2014-08-22 19:29:34,220.220 INFO:tasks.rados.rados.0.mira097.stdout:1405: done (0 left) 2014-08-22 19:29:34,220.220 INFO:tasks.rados.rados.0.mira097.stdout:1406: read oid 96 snap -1 2014-08-22 19:29:34,220.220 INFO:tasks.rados.rados.0.mira097.stdout:1407: snap_create 2014-08-22 19:29:34,221.221 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:34,237.237 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.3 to 1 (8655362) 2014-08-22 19:29:34,243.243 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.145203 7f5a3ce1b700 20 client.4232.objecter shutdown clearing up homeless session... 2014-08-22 19:29:34,244.244 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.146066 7f5a3ce1b700 1 librados: shutdown 2014-08-22 19:29:34,252.252 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:29:34,281.281 INFO:tasks.rados.rados.0.mira097.stdout:1406: expect (ObjNum 95 snap 0 seq_num 95) 2014-08-22 19:29:34,698.698 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.600194 7f6b9d3d8700 1 librados: starting msgr at :/0 2014-08-22 19:29:34,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.600201 7f6b9d3d8700 1 librados: starting objecter 2014-08-22 19:29:34,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.600353 7f6b9d3d8700 1 librados: setting wanted keys 2014-08-22 19:29:34,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.600358 7f6b9d3d8700 1 librados: calling monclient init 2014-08-22 19:29:34,699.699 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.602044 7f6b962e8700 10 client.?.objecter ms_handle_connect 0x7f6b980355e0 2014-08-22 19:29:34,700.700 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.602081 7f6b962e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:34,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.605412 7f6b9d3d8700 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:34,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.605551 7f6b9d3d8700 1 librados: init done 2014-08-22 19:29:34,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606312 7f6b962e8700 10 client.4261.objecter ms_dispatch 0x7f6b98000f80 osd_map(237..237 src has 1..237) v3 2014-08-22 19:29:34,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606324 7f6b962e8700 3 client.4261.objecter handle_osd_map got epochs [237,237] > 0 2014-08-22 19:29:34,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606327 7f6b962e8700 3 client.4261.objecter handle_osd_map decoding full epoch 237 2014-08-22 19:29:34,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606422 7f6b962e8700 20 client.4261.objecter dump_active .. 0 homeless 2014-08-22 19:29:34,705.705 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606657 7f6b962e8700 10 client.4261.objecter ms_dispatch 0x7f6b98000f80 osd_map(237..237 src has 1..237) v3 2014-08-22 19:29:34,800.800 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606661 7f6b962e8700 3 client.4261.objecter handle_osd_map ignoring epochs [237,237] <= 237 2014-08-22 19:29:34,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:27.606663 7f6b962e8700 20 client.4261.objecter dump_active .. 0 homeless 2014-08-22 19:29:35,247.247 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:29:35,250.250 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.151842 7f6b9d3d8700 20 client.4261.objecter shutdown clearing up homeless session... 2014-08-22 19:29:35,250.250 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.152166 7f6b9d3d8700 1 librados: shutdown 2014-08-22 19:29:35,254.254 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:29:35,386.386 INFO:tasks.rados.rados.0.mira097.stdout:1406: done (0 left) 2014-08-22 19:29:35,833.833 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.734561 7ffca5e43700 1 librados: starting msgr at :/0 2014-08-22 19:29:35,833.833 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.734578 7ffca5e43700 1 librados: starting objecter 2014-08-22 19:29:35,833.833 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.734801 7ffca5e43700 1 librados: setting wanted keys 2014-08-22 19:29:35,833.833 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.734810 7ffca5e43700 1 librados: calling monclient init 2014-08-22 19:29:35,834.834 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.736020 7ffc9e7fc700 10 client.?.objecter ms_handle_connect 0x7ffca00355e0 2014-08-22 19:29:35,834.834 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.736062 7ffc9e7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:35,836.836 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739148 7ffca5e43700 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:35,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739324 7ffca5e43700 1 librados: init done 2014-08-22 19:29:35,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739723 7ffc9e7fc700 10 client.4235.objecter ms_dispatch 0x7ffca0000f80 osd_map(238..238 src has 1..238) v3 2014-08-22 19:29:35,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739735 7ffc9e7fc700 3 client.4235.objecter handle_osd_map got epochs [238,238] > 0 2014-08-22 19:29:35,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739739 7ffc9e7fc700 3 client.4235.objecter handle_osd_map decoding full epoch 238 2014-08-22 19:29:35,914.914 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739895 7ffc9e7fc700 20 client.4235.objecter dump_active .. 0 homeless 2014-08-22 19:29:35,914.914 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739979 7ffc9e7fc700 10 client.4235.objecter ms_dispatch 0x7ffca0000f80 osd_map(238..238 src has 1..238) v3 2014-08-22 19:29:35,915.915 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739991 7ffc9e7fc700 3 client.4235.objecter handle_osd_map ignoring epochs [238,238] <= 238 2014-08-22 19:29:35,915.915 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:28.739998 7ffc9e7fc700 20 client.4235.objecter dump_active .. 0 homeless 2014-08-22 19:29:36,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1407: done (0 left) 2014-08-22 19:29:36,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1408: write oid 386 current snap is 91 2014-08-22 19:29:36,257.257 INFO:tasks.rados.rados.0.mira097.stdout:1408: seq_num 822 ranges {713280=407969,1877288=709001,2713279=1} 2014-08-22 19:29:36,282.282 INFO:tasks.rados.rados.0.mira097.stdout:1408: writing mira09725123-386 from 713280 to 1121249 tid 1 2014-08-22 19:29:36,283.283 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:29:36,286.286 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.188523 7ffca5e43700 20 client.4235.objecter shutdown clearing up homeless session... 2014-08-22 19:29:36,287.287 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.188991 7ffca5e43700 1 librados: shutdown 2014-08-22 19:29:36,291.291 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:29:36,309.309 INFO:tasks.rados.rados.0.mira097.stdout:1408: writing mira09725123-386 from 1877288 to 2586289 tid 2 2014-08-22 19:29:36,312.312 INFO:tasks.rados.rados.0.mira097.stdout:1408: writing mira09725123-386 from 2713279 to 2713280 tid 3 2014-08-22 19:29:36,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1409: snap_create 2014-08-22 19:29:36,315.315 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:36,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1408: finishing write tid 1 to mira09725123-386 2014-08-22 19:29:36,335.335 INFO:tasks.rados.rados.0.mira097.stdout:1408: finishing write tid 2 to mira09725123-386 2014-08-22 19:29:36,335.335 INFO:tasks.rados.rados.0.mira097.stdout:1408: finishing write tid 3 to mira09725123-386 2014-08-22 19:29:36,428.428 INFO:tasks.rados.rados.0.mira097.stdout:1408: finishing write tid 5 to mira09725123-386 2014-08-22 19:29:36,428.428 INFO:tasks.rados.rados.0.mira097.stdout:1408: finishing write tid 6 to mira09725123-386 2014-08-22 19:29:36,428.428 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 386 v 1901 (ObjNum 822 snap 91 seq_num 822) dirty exists 2014-08-22 19:29:36,429.429 INFO:tasks.rados.rados.0.mira097.stdout:1408: done (0 left) 2014-08-22 19:29:36,795.795 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.696796 7f1082748700 1 librados: starting msgr at :/0 2014-08-22 19:29:36,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.696804 7f1082748700 1 librados: starting objecter 2014-08-22 19:29:36,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.697098 7f1082748700 1 librados: setting wanted keys 2014-08-22 19:29:36,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.697114 7f1082748700 1 librados: calling monclient init 2014-08-22 19:29:36,797.797 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.698850 7f107affd700 10 client.?.objecter ms_handle_connect 0x7f107c0355e0 2014-08-22 19:29:36,797.797 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.698889 7f107affd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:36,800.800 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702030 7f1082748700 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:36,800.800 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702173 7f1082748700 1 librados: init done 2014-08-22 19:29:36,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702685 7f107affd700 10 client.4257.objecter ms_dispatch 0x7f107c000f80 osd_map(239..239 src has 1..239) v3 2014-08-22 19:29:36,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702707 7f107affd700 3 client.4257.objecter handle_osd_map got epochs [239,239] > 0 2014-08-22 19:29:36,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702713 7f107affd700 3 client.4257.objecter handle_osd_map decoding full epoch 239 2014-08-22 19:29:36,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702862 7f107affd700 20 client.4257.objecter dump_active .. 0 homeless 2014-08-22 19:29:36,801.801 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702916 7f107affd700 10 client.4257.objecter ms_dispatch 0x7f107c000f80 osd_map(239..239 src has 1..239) v3 2014-08-22 19:29:36,802.802 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702922 7f107affd700 3 client.4257.objecter handle_osd_map ignoring epochs [239,239] <= 239 2014-08-22 19:29:36,802.802 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:29.702924 7f107affd700 20 client.4257.objecter dump_active .. 0 homeless 2014-08-22 19:29:37,269.269 INFO:tasks.rados.rados.0.mira097.stdout:1409: done (0 left) 2014-08-22 19:29:37,269.269 INFO:tasks.rados.rados.0.mira097.stdout:1410: rollback oid 403 current snap is 92 2014-08-22 19:29:37,269.269 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 403 to 88 2014-08-22 19:29:37,270.270 INFO:tasks.rados.rados.0.mira097.stdout:1411: snap_create 2014-08-22 19:29:37,270.270 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:37,305.305 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:29:37,309.309 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.210605 7f1082748700 20 client.4257.objecter shutdown clearing up homeless session... 2014-08-22 19:29:37,310.310 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.211072 7f1082748700 1 librados: shutdown 2014-08-22 19:29:37,317.317 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 5 1' 2014-08-22 19:29:37,334.334 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 403 v 1942 (ObjNum 402 snap 0 seq_num 402) dirty exists 2014-08-22 19:29:37,334.334 INFO:tasks.rados.rados.0.mira097.stdout:1410: done (0 left) 2014-08-22 19:29:37,790.790 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.691302 7f29d3bfd700 1 librados: starting msgr at :/0 2014-08-22 19:29:37,790.790 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.691308 7f29d3bfd700 1 librados: starting objecter 2014-08-22 19:29:37,790.790 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.691623 7f29d3bfd700 1 librados: setting wanted keys 2014-08-22 19:29:37,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.691628 7f29d3bfd700 1 librados: calling monclient init 2014-08-22 19:29:37,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.693307 7f29d0caa700 10 client.?.objecter ms_handle_connect 0x7f29cc0355e0 2014-08-22 19:29:37,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.693334 7f29d0caa700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:37,794.794 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.696831 7f29d3bfd700 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:37,795.795 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.696964 7f29d3bfd700 1 librados: init done 2014-08-22 19:29:37,795.795 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.697847 7f29d0caa700 10 client.4264.objecter ms_dispatch 0x7f29cc000f80 osd_map(240..240 src has 1..240) v3 2014-08-22 19:29:37,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.697861 7f29d0caa700 3 client.4264.objecter handle_osd_map got epochs [240,240] > 0 2014-08-22 19:29:37,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.697865 7f29d0caa700 3 client.4264.objecter handle_osd_map decoding full epoch 240 2014-08-22 19:29:37,796.796 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.698008 7f29d0caa700 20 client.4264.objecter dump_active .. 0 homeless 2014-08-22 19:29:37,868.868 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.698063 7f29d0caa700 10 client.4264.objecter ms_dispatch 0x7f29cc000f80 osd_map(240..240 src has 1..240) v3 2014-08-22 19:29:37,869.869 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.698068 7f29d0caa700 3 client.4264.objecter handle_osd_map ignoring epochs [240,240] <= 240 2014-08-22 19:29:37,869.869 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:30.698070 7f29d0caa700 20 client.4264.objecter dump_active .. 0 homeless 2014-08-22 19:29:38,311.311 INFO:tasks.rados.rados.0.mira097.stdout:1411: done (0 left) 2014-08-22 19:29:38,311.311 INFO:tasks.rados.rados.0.mira097.stdout:1412: read oid 441 snap -1 2014-08-22 19:29:38,311.311 INFO:tasks.rados.rados.0.mira097.stdout:1413: snap_create 2014-08-22 19:29:38,311.311 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:38,343.343 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.5 to 1 (8655362) 2014-08-22 19:29:38,347.347 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.248164 7f29d3bfd700 20 client.4264.objecter shutdown clearing up homeless session... 2014-08-22 19:29:38,347.347 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.248899 7f29d3bfd700 1 librados: shutdown 2014-08-22 19:29:38,353.353 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:29:38,353.353 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:29:38,397.397 INFO:tasks.rados.rados.0.mira097.stdout:1412: expect (ObjNum 440 snap 0 seq_num 440) 2014-08-22 19:29:38,819.819 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.720030 7f39cdd3f700 1 librados: starting msgr at :/0 2014-08-22 19:29:38,819.819 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.720118 7f39cdd3f700 1 librados: starting objecter 2014-08-22 19:29:38,819.819 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.720284 7f39cdd3f700 1 librados: setting wanted keys 2014-08-22 19:29:38,820.820 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.720288 7f39cdd3f700 1 librados: calling monclient init 2014-08-22 19:29:38,820.820 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.721068 7f39c67fc700 10 client.?.objecter ms_handle_connect 0x7f39c80355e0 2014-08-22 19:29:38,820.820 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.721102 7f39c67fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:38,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722449 7f39cdd3f700 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:38,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722554 7f39cdd3f700 1 librados: init done 2014-08-22 19:29:38,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722786 7f39c67fc700 10 client.4238.objecter ms_dispatch 0x7f39c8000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:38,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722794 7f39c67fc700 3 client.4238.objecter handle_osd_map got epochs [241,241] > 0 2014-08-22 19:29:38,821.821 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722797 7f39c67fc700 3 client.4238.objecter handle_osd_map decoding full epoch 241 2014-08-22 19:29:38,822.822 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722887 7f39c67fc700 20 client.4238.objecter dump_active .. 0 homeless 2014-08-22 19:29:38,822.822 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722916 7f39c67fc700 10 client.4238.objecter ms_dispatch 0x7f39c8000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:38,822.822 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722918 7f39c67fc700 3 client.4238.objecter handle_osd_map ignoring epochs [241,241] <= 241 2014-08-22 19:29:38,822.822 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:31.722919 7f39c67fc700 20 client.4238.objecter dump_active .. 0 homeless 2014-08-22 19:29:38,906.906 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:29:39,348.348 INFO:tasks.rados.rados.0.mira097.stdout:1412: done (0 left) 2014-08-22 19:29:39,349.349 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:32.080919 7f39cdd3f700 20 client.4238.objecter shutdown clearing up homeless session... 2014-08-22 19:29:39,349.349 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:32.081480 7f39cdd3f700 1 librados: shutdown 2014-08-22 19:29:39,542.542 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:29:40,149.149 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.049760 7fb0b5b0e700 1 librados: starting msgr at :/0 2014-08-22 19:29:40,150.150 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.049765 7fb0b5b0e700 1 librados: starting objecter 2014-08-22 19:29:40,150.150 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.050132 7fb0b5b0e700 1 librados: setting wanted keys 2014-08-22 19:29:40,150.150 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.050136 7fb0b5b0e700 1 librados: calling monclient init 2014-08-22 19:29:40,150.150 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.051060 7fb0ae7fc700 10 client.?.objecter ms_handle_connect 0x7fb0b00355e0 2014-08-22 19:29:40,150.150 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.051101 7fb0ae7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:40,151.151 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.053919 7fb0b5b0e700 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:40,152.152 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054117 7fb0b5b0e700 1 librados: init done 2014-08-22 19:29:40,152.152 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054437 7fb0ae7fc700 10 client.4241.objecter ms_dispatch 0x7fb0b0000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054449 7fb0ae7fc700 3 client.4241.objecter handle_osd_map got epochs [241,241] > 0 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054452 7fb0ae7fc700 3 client.4241.objecter handle_osd_map decoding full epoch 241 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054639 7fb0ae7fc700 20 client.4241.objecter dump_active .. 0 homeless 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054706 7fb0ae7fc700 10 client.4241.objecter ms_dispatch 0x7fb0b0000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054711 7fb0ae7fc700 3 client.4241.objecter handle_osd_map ignoring epochs [241,241] <= 241 2014-08-22 19:29:40,153.153 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.054714 7fb0ae7fc700 20 client.4241.objecter dump_active .. 0 homeless 2014-08-22 19:29:40,268.268 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:29:40,500.500 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.348440 7fb0b5b0e700 20 client.4241.objecter shutdown clearing up homeless session... 2014-08-22 19:29:40,500.500 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:33.349422 7fb0b5b0e700 1 librados: shutdown 2014-08-22 19:29:40,637.637 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:29:41,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.208273 7f1317784700 1 librados: starting msgr at :/0 2014-08-22 19:29:41,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.208287 7f1317784700 1 librados: starting objecter 2014-08-22 19:29:41,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.208525 7f1317784700 1 librados: setting wanted keys 2014-08-22 19:29:41,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.208532 7f1317784700 1 librados: calling monclient init 2014-08-22 19:29:41,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.209493 7f1314831700 10 client.?.objecter ms_handle_connect 0x7f13100355e0 2014-08-22 19:29:41,309.309 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:34.209590 7f1314831700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:44,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:37.209534 7f1314831700 10 client.?.objecter ms_handle_connect 0x7f12fc000e90 2014-08-22 19:29:44,308.308 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:37.209545 7f1314831700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:45,434.434 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.335605 7f1317784700 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:45,434.434 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.336649 7f1317784700 1 librados: init done 2014-08-22 19:29:45,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337136 7f1314831700 10 client.4244.objecter ms_dispatch 0x7f1310000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:45,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337148 7f1314831700 3 client.4244.objecter handle_osd_map got epochs [241,241] > 0 2014-08-22 19:29:45,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337152 7f1314831700 3 client.4244.objecter handle_osd_map decoding full epoch 241 2014-08-22 19:29:45,436.436 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337297 7f1314831700 20 client.4244.objecter dump_active .. 0 homeless 2014-08-22 19:29:45,436.436 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337357 7f1314831700 10 client.4244.objecter ms_dispatch 0x7f1310000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:45,436.436 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337362 7f1314831700 3 client.4244.objecter handle_osd_map ignoring epochs [241,241] <= 241 2014-08-22 19:29:45,436.436 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.337364 7f1314831700 20 client.4244.objecter dump_active .. 0 homeless 2014-08-22 19:29:45,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.480666 7f1317784700 20 client.4244.objecter shutdown clearing up homeless session... 2014-08-22 19:29:45,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:38.481324 7f1317784700 1 librados: shutdown 2014-08-22 19:29:45,609.609 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003310', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001678', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:29:36.480850', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873786500, u'health': u'HEALTH_OK', u'kb_used': 22994552, u'store_stats': {u'bytes_total': 12512039, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316148}}, {u'last_updated': u'2014-08-22 11:29:36.470781', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 876865576, u'health': u'HEALTH_OK', u'kb_used': 19915476, u'store_stats': {u'bytes_total': 13560624, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316157}}, {u'last_updated': u'2014-08-22 11:29:36.073860', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873786520, u'health': u'HEALTH_OK', u'kb_used': 22994532, u'store_stats': {u'bytes_total': 13560624, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 67123, u'bytes_sst': 8316157}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'mon.c has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 4998582538240, u'num_pgs': 92, u'data_bytes': 1223151597, u'bytes_used': 3265642496, u'op_per_sec': 1, u'version': 397, u'pgs_by_state': [{u'count': 92, u'state_name': u'active+clean'}], u'read_bytes_sec': 462021, u'write_bytes_sec': 177251, u'bytes_avail': 4995316895744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 241, u'num_in_osds': 5}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:29:45,610.610 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:29:45,610.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:45,610.610 INFO:tasks.thrashosds.thrasher:Adding osd 1 2014-08-22 19:29:45,610.610 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1' 2014-08-22 19:29:46,277.277 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.178565 7fae0d1b0700 1 librados: starting msgr at :/0 2014-08-22 19:29:46,277.277 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.178573 7fae0d1b0700 1 librados: starting objecter 2014-08-22 19:29:46,277.277 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.178871 7fae0d1b0700 1 librados: setting wanted keys 2014-08-22 19:29:46,278.278 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.178885 7fae0d1b0700 1 librados: calling monclient init 2014-08-22 19:29:46,278.278 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.180286 7fae060ae700 10 client.?.objecter ms_handle_connect 0x7fae080355e0 2014-08-22 19:29:46,279.279 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.180338 7fae060ae700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:46,281.281 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.183589 7fae0d1b0700 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:46,282.282 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.183787 7fae0d1b0700 1 librados: init done 2014-08-22 19:29:46,283.283 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184321 7fae060ae700 10 client.4247.objecter ms_dispatch 0x7fae08000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:46,283.283 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184339 7fae060ae700 3 client.4247.objecter handle_osd_map got epochs [241,241] > 0 2014-08-22 19:29:46,283.283 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184345 7fae060ae700 3 client.4247.objecter handle_osd_map decoding full epoch 241 2014-08-22 19:29:46,284.284 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184554 7fae060ae700 20 client.4247.objecter dump_active .. 0 homeless 2014-08-22 19:29:46,284.284 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184616 7fae060ae700 10 client.4247.objecter ms_dispatch 0x7fae08000f80 osd_map(241..241 src has 1..241) v3 2014-08-22 19:29:46,284.284 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184623 7fae060ae700 3 client.4247.objecter handle_osd_map ignoring epochs [241,241] <= 241 2014-08-22 19:29:46,284.284 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.184627 7fae060ae700 20 client.4247.objecter dump_active .. 0 homeless 2014-08-22 19:29:46,414.414 INFO:tasks.rados.rados.0.mira097.stdout:1413: done (0 left) 2014-08-22 19:29:46,414.414 INFO:tasks.rados.rados.0.mira097.stdout:1414: snap_remove snap 85 2014-08-22 19:29:46,435.435 INFO:teuthology.orchestra.run.mira097.stderr:marked in osd.1. 2014-08-22 19:29:46,435.435 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.337183 7fae0d1b0700 20 client.4247.objecter shutdown clearing up homeless session... 2014-08-22 19:29:46,449.449 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:39.339587 7fae0d1b0700 1 librados: shutdown 2014-08-22 19:29:46,450.450 INFO:tasks.thrashosds.thrasher:Added osd 1 2014-08-22 19:29:47,420.420 INFO:tasks.rados.rados.0.mira097.stdout:1414: done (0 left) 2014-08-22 19:29:47,420.420 INFO:tasks.rados.rados.0.mira097.stdout:1415: write oid 257 current snap is 94 2014-08-22 19:29:47,420.420 INFO:tasks.rados.rados.0.mira097.stdout:1415: seq_num 823 ranges {409533=531197,1591419=530620,2409532=1} 2014-08-22 19:29:47,447.447 INFO:tasks.rados.rados.0.mira097.stdout:1415: writing mira09725123-257 from 409533 to 940730 tid 1 2014-08-22 19:29:47,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1415: writing mira09725123-257 from 1591419 to 2122039 tid 2 2014-08-22 19:29:47,476.476 INFO:tasks.rados.rados.0.mira097.stdout:1415: writing mira09725123-257 from 2409532 to 2409533 tid 3 2014-08-22 19:29:47,478.478 INFO:tasks.rados.rados.0.mira097.stdout:1416: snap_create 2014-08-22 19:29:47,478.478 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:47,478.478 INFO:tasks.rados.rados.0.mira097.stdout:1415: finishing write tid 1 to mira09725123-257 2014-08-22 19:29:47,498.498 INFO:tasks.rados.rados.0.mira097.stdout:1415: finishing write tid 2 to mira09725123-257 2014-08-22 19:29:47,498.498 INFO:tasks.rados.rados.0.mira097.stdout:1415: finishing write tid 3 to mira09725123-257 2014-08-22 19:29:47,498.498 INFO:tasks.rados.rados.0.mira097.stdout:1415: finishing write tid 5 to mira09725123-257 2014-08-22 19:29:47,499.499 INFO:tasks.rados.rados.0.mira097.stdout:1415: finishing write tid 6 to mira09725123-257 2014-08-22 19:29:47,499.499 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 257 v 1868 (ObjNum 823 snap 94 seq_num 823) dirty exists 2014-08-22 19:29:47,499.499 INFO:tasks.rados.rados.0.mira097.stdout:1415: done (0 left) 2014-08-22 19:29:48,488.488 INFO:tasks.rados.rados.0.mira097.stdout:1416: done (0 left) 2014-08-22 19:29:48,489.489 INFO:tasks.rados.rados.0.mira097.stdout:1417: read oid 256 snap -1 2014-08-22 19:29:48,489.489 INFO:tasks.rados.rados.0.mira097.stdout:1418: copy_from oid 265 from oid 263 current snap is 95 2014-08-22 19:29:48,489.489 INFO:tasks.rados.rados.0.mira097.stdout:1419: snap_create 2014-08-22 19:29:48,490.490 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:29:48,498.498 INFO:tasks.rados.rados.0.mira097.stdout:1417: expect (ObjNum 703 snap 52 seq_num 703) 2014-08-22 19:29:49,511.511 INFO:tasks.rados.rados.0.mira097.stdout:1417: done (1 left) 2014-08-22 19:29:49,512.512 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:29:51,450.450 INFO:tasks.thrashosds.thrasher:in_osds: [3, 2, 0, 4, 5, 1] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:51,450.450 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:51,450.450 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [3, 2, 0, 4, 5, 1] 2014-08-22 19:29:51,451.451 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3' 2014-08-22 19:29:51,923.923 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.825095 7f01f7b58700 1 librados: starting msgr at :/0 2014-08-22 19:29:51,924.924 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.825102 7f01f7b58700 1 librados: starting objecter 2014-08-22 19:29:51,924.924 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.825345 7f01f7b58700 1 librados: setting wanted keys 2014-08-22 19:29:51,924.924 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.825353 7f01f7b58700 1 librados: calling monclient init 2014-08-22 19:29:51,924.924 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.827065 7f01f4c05700 10 client.?.objecter ms_handle_connect 0x7f01f00355e0 2014-08-22 19:29:51,925.925 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.827183 7f01f4c05700 10 client.?.objecter resend_mon_ops 2014-08-22 19:29:51,928.928 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.830214 7f01f7b58700 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:29:51,928.928 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.830381 7f01f7b58700 1 librados: init done 2014-08-22 19:29:51,932.932 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.830870 7f01f4c05700 10 client.4260.objecter ms_dispatch 0x7f01f0000f80 osd_map(244..244 src has 1..244) v3 2014-08-22 19:29:52,001.001 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.830886 7f01f4c05700 3 client.4260.objecter handle_osd_map got epochs [244,244] > 0 2014-08-22 19:29:52,002.002 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.830891 7f01f4c05700 3 client.4260.objecter handle_osd_map decoding full epoch 244 2014-08-22 19:29:52,002.002 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.831017 7f01f4c05700 20 client.4260.objecter dump_active .. 0 homeless 2014-08-22 19:29:52,002.002 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.831055 7f01f4c05700 10 client.4260.objecter ms_dispatch 0x7f01f0000f80 osd_map(244..244 src has 1..244) v3 2014-08-22 19:29:52,002.002 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.831058 7f01f4c05700 3 client.4260.objecter handle_osd_map ignoring epochs [244,244] <= 244 2014-08-22 19:29:52,002.002 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:44.831059 7f01f4c05700 20 client.4260.objecter dump_active .. 0 homeless 2014-08-22 19:29:52,784.784 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.3. 2014-08-22 19:29:52,787.787 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:45.689304 7f01f7b58700 20 client.4260.objecter shutdown clearing up homeless session... 2014-08-22 19:29:52,787.787 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:45.689793 7f01f7b58700 1 librados: shutdown 2014-08-22 19:29:57,796.796 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 5, 1] out_osds: [3] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:29:57,796.796 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:29:57,796.796 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2014-08-22 19:29:57,797.797 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:29:57,797.797 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:29:57,797.797 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-08-22 19:30:03,284.284 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 5, 1] out_osds: [3] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:03,285.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:03,285.285 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd 2014-08-22 19:30:03,285.285 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:30:03,674.674 INFO:tasks.rados.rados.0.mira097.stdout:1418: finishing copy_from to mira09725123-265 2014-08-22 19:30:03,675.675 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 265 v 1756 (ObjNum 719 snap 56 seq_num 719) dirty exists 2014-08-22 19:30:03,675.675 INFO:tasks.rados.rados.0.mira097.stdout:1418: finishing copy_from racing read to mira09725123-265 2014-08-22 19:30:03,675.675 INFO:tasks.rados.rados.0.mira097.stdout:1418: done (0 left) 2014-08-22 19:30:03,766.766 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.667340 7f5ac0493700 1 librados: starting msgr at :/0 2014-08-22 19:30:03,766.766 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.667359 7f5ac0493700 1 librados: starting objecter 2014-08-22 19:30:03,766.766 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.667730 7f5ac0493700 1 librados: setting wanted keys 2014-08-22 19:30:03,766.766 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.667752 7f5ac0493700 1 librados: calling monclient init 2014-08-22 19:30:03,767.767 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.669385 7f5abd540700 10 client.?.objecter ms_handle_connect 0x7f5ab80355e0 2014-08-22 19:30:03,767.767 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.669430 7f5abd540700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:03,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.672410 7f5ac0493700 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:03,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.672586 7f5ac0493700 1 librados: init done 2014-08-22 19:30:03,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.672983 7f5abd540700 10 client.4250.objecter ms_dispatch 0x7f5ab8000f80 osd_map(246..246 src has 1..246) v3 2014-08-22 19:30:03,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.672999 7f5abd540700 3 client.4250.objecter handle_osd_map got epochs [246,246] > 0 2014-08-22 19:30:03,837.837 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.673003 7f5abd540700 3 client.4250.objecter handle_osd_map decoding full epoch 246 2014-08-22 19:30:03,838.838 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.673190 7f5abd540700 20 client.4250.objecter dump_active .. 0 homeless 2014-08-22 19:30:03,838.838 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.673237 7f5abd540700 10 client.4250.objecter ms_dispatch 0x7f5ab8000f80 osd_map(246..246 src has 1..246) v3 2014-08-22 19:30:03,838.838 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.673242 7f5abd540700 3 client.4250.objecter handle_osd_map ignoring epochs [246,246] <= 246 2014-08-22 19:30:03,838.838 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:56.673244 7f5abd540700 20 client.4250.objecter dump_active .. 0 homeless 2014-08-22 19:30:03,946.946 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:30:04,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1419: done (0 left) 2014-08-22 19:30:04,237.237 INFO:tasks.rados.rados.0.mira097.stdout:1420: copy_from oid 366 from oid 338 current snap is 96 2014-08-22 19:30:04,238.238 INFO:tasks.rados.rados.0.mira097.stdout:1421: snap_create 2014-08-22 19:30:04,238.238 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:04,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1420: finishing copy_from to mira09725123-366 2014-08-22 19:30:04,251.251 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 366 v 429 (ObjNum 337 snap 0 seq_num 337) dirty exists 2014-08-22 19:30:04,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1420: finishing copy_from racing read to mira09725123-366 2014-08-22 19:30:04,251.251 INFO:tasks.rados.rados.0.mira097.stdout:1420: done (0 left) 2014-08-22 19:30:04,286.286 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.025189 7f5ac0493700 20 client.4250.objecter shutdown clearing up homeless session... 2014-08-22 19:30:04,286.286 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.025873 7f5ac0493700 1 librados: shutdown 2014-08-22 19:30:04,434.434 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 84' 2014-08-22 19:30:05,049.049 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.950442 7fcf93680700 1 librados: starting msgr at :/0 2014-08-22 19:30:05,050.050 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.950453 7fcf93680700 1 librados: starting objecter 2014-08-22 19:30:05,050.050 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.950719 7fcf93680700 1 librados: setting wanted keys 2014-08-22 19:30:05,050.050 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.950733 7fcf93680700 1 librados: calling monclient init 2014-08-22 19:30:05,051.051 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.952092 7fcf8bfff700 10 client.?.objecter ms_handle_connect 0x7fcf8c0355e0 2014-08-22 19:30:05,051.051 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.952161 7fcf8bfff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:05,053.053 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.954788 7fcf93680700 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:05,053.053 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.954970 7fcf93680700 1 librados: init done 2014-08-22 19:30:05,054.054 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955706 7fcf8bfff700 10 client.4253.objecter ms_dispatch 0x7fcf8c000f80 osd_map(247..247 src has 1..247) v3 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955721 7fcf8bfff700 3 client.4253.objecter handle_osd_map got epochs [247,247] > 0 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955725 7fcf8bfff700 3 client.4253.objecter handle_osd_map decoding full epoch 247 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955921 7fcf8bfff700 20 client.4253.objecter dump_active .. 0 homeless 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955982 7fcf8bfff700 10 client.4253.objecter ms_dispatch 0x7fcf8c000f80 osd_map(247..247 src has 1..247) v3 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955987 7fcf8bfff700 3 client.4253.objecter handle_osd_map ignoring epochs [247,247] <= 247 2014-08-22 19:30:05,055.055 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:57.955990 7fcf8bfff700 20 client.4253.objecter dump_active .. 0 homeless 2014-08-22 19:30:05,171.171 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pgp_num to 84 2014-08-22 19:30:05,188.188 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:58.075862 7fcf93680700 20 client.4253.objecter shutdown clearing up homeless session... 2014-08-22 19:30:05,188.188 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:29:58.079964 7fcf93680700 1 librados: shutdown 2014-08-22 19:30:05,188.188 INFO:tasks.rados.rados.0.mira097.stdout:1421: done (0 left) 2014-08-22 19:30:05,189.189 INFO:tasks.rados.rados.0.mira097.stdout:1422: copy_from oid 47 from oid 28 current snap is 97 2014-08-22 19:30:05,189.189 INFO:tasks.rados.rados.0.mira097.stdout:1423: snap_create 2014-08-22 19:30:05,189.189 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:08,739.739 INFO:tasks.rados.rados.0.mira097.stdout:1422: finishing copy_from to mira09725123-47 2014-08-22 19:30:08,740.740 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 47 v 41 (ObjNum 27 snap 0 seq_num 27) dirty exists 2014-08-22 19:30:08,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1422: finishing copy_from racing read to mira09725123-47 2014-08-22 19:30:08,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1422: done (0 left) 2014-08-22 19:30:08,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1423: done (0 left) 2014-08-22 19:30:08,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1424: snap_create 2014-08-22 19:30:09,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1424: done (0 left) 2014-08-22 19:30:09,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1425: copy_from oid 235 from oid 73 current snap is 99 2014-08-22 19:30:09,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1426: read oid 13 snap 96 2014-08-22 19:30:09,886.886 INFO:tasks.rados.rados.0.mira097.stdout:1427: snap_remove snap 89 2014-08-22 19:30:09,957.957 INFO:tasks.rados.rados.0.mira097.stdout:1426: expect (ObjNum 739 snap 63 seq_num 739) 2014-08-22 19:30:10,185.185 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 5, 1] out_osds: [3] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:10,186.186 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:10,186.186 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [2, 0, 4, 5, 1] 2014-08-22 19:30:10,186.186 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-08-22 19:30:10,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.554517 7f3bd6cd6700 1 librados: starting msgr at :/0 2014-08-22 19:30:10,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.554524 7f3bd6cd6700 1 librados: starting objecter 2014-08-22 19:30:10,653.653 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.554676 7f3bd6cd6700 1 librados: setting wanted keys 2014-08-22 19:30:10,654.654 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.554683 7f3bd6cd6700 1 librados: calling monclient init 2014-08-22 19:30:10,654.654 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.556408 7f3bc77fe700 10 client.?.objecter ms_handle_connect 0x7f3bd00355e0 2014-08-22 19:30:10,655.655 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.556427 7f3bc77fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:10,658.658 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.560329 7f3bd6cd6700 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:10,658.658 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.560476 7f3bd6cd6700 1 librados: init done 2014-08-22 19:30:10,659.659 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561494 7f3bc77fe700 10 client.4267.objecter ms_dispatch 0x7f3bd0000f80 osd_map(250..250 src has 1..250) v3 2014-08-22 19:30:10,660.660 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561501 7f3bc77fe700 3 client.4267.objecter handle_osd_map got epochs [250,250] > 0 2014-08-22 19:30:10,660.660 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561503 7f3bc77fe700 3 client.4267.objecter handle_osd_map decoding full epoch 250 2014-08-22 19:30:10,660.660 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561592 7f3bc77fe700 20 client.4267.objecter dump_active .. 0 homeless 2014-08-22 19:30:10,661.661 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561749 7f3bc77fe700 10 client.4267.objecter ms_dispatch 0x7f3bd0000f80 osd_map(250..250 src has 1..250) v3 2014-08-22 19:30:10,661.661 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561756 7f3bc77fe700 3 client.4267.objecter handle_osd_map ignoring epochs [250,250] <= 250 2014-08-22 19:30:10,661.661 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:03.561758 7f3bc77fe700 20 client.4267.objecter dump_active .. 0 homeless 2014-08-22 19:30:11,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1425: finishing copy_from to mira09725123-235 2014-08-22 19:30:11,987.987 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 235 v 618 (ObjNum 454 snap 0 seq_num 454) dirty exists 2014-08-22 19:30:11,988.988 INFO:tasks.rados.rados.0.mira097.stdout:1425: finishing copy_from racing read to mira09725123-235 2014-08-22 19:30:11,991.991 INFO:tasks.rados.rados.0.mira097.stdout:1425: done (2 left) 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1426: done (1 left) 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1427: done (0 left) 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1428: read oid 344 snap 84 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1429: read oid 69 snap 87 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1430: copy_from oid 133 from oid 404 current snap is 99 2014-08-22 19:30:11,992.992 INFO:tasks.rados.rados.0.mira097.stdout:1431: write oid 458 current snap is 99 2014-08-22 19:30:11,993.993 INFO:tasks.rados.rados.0.mira097.stdout:1431: seq_num 824 ranges {754363=504026,1671504=600815,2784445=369918} 2014-08-22 19:30:12,005.005 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.5. 2014-08-22 19:30:12,009.009 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:04.910067 7f3bd6cd6700 20 client.4267.objecter shutdown clearing up homeless session... 2014-08-22 19:30:12,160.160 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:04.910346 7f3bd6cd6700 1 librados: shutdown 2014-08-22 19:30:12,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1431: writing mira09725123-458 from 754363 to 1258389 tid 1 2014-08-22 19:30:12,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1431: writing mira09725123-458 from 1671504 to 2272319 tid 2 2014-08-22 19:30:12,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1431: writing mira09725123-458 from 2784445 to 3154363 tid 3 2014-08-22 19:30:12,161.161 INFO:tasks.rados.rados.0.mira097.stdout:1432: snap_remove snap 95 2014-08-22 19:30:13,509.509 INFO:tasks.rados.rados.0.mira097.stdout:1428: done (4 left) 2014-08-22 19:30:13,510.510 INFO:tasks.rados.rados.0.mira097.stdout:1432: done (3 left) 2014-08-22 19:30:13,510.510 INFO:tasks.rados.rados.0.mira097.stdout:1433: snap_remove snap 83 2014-08-22 19:30:13,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1429: expect (ObjNum 768 snap 70 seq_num 768) 2014-08-22 19:30:13,777.777 INFO:tasks.rados.rados.0.mira097.stdout:1431: finishing write tid 1 to mira09725123-458 2014-08-22 19:30:13,778.778 INFO:tasks.rados.rados.0.mira097.stdout:1431: finishing write tid 2 to mira09725123-458 2014-08-22 19:30:13,778.778 INFO:tasks.rados.rados.0.mira097.stdout:1431: finishing write tid 3 to mira09725123-458 2014-08-22 19:30:13,778.778 INFO:tasks.rados.rados.0.mira097.stdout:1431: finishing write tid 5 to mira09725123-458 2014-08-22 19:30:13,784.784 INFO:tasks.rados.rados.0.mira097.stdout:1430: finishing copy_from to mira09725123-133 2014-08-22 19:30:13,784.784 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 133 v 1691 (ObjNum 736 snap 62 seq_num 736) dirty exists 2014-08-22 19:30:13,805.805 INFO:tasks.rados.rados.0.mira097.stdout:1431: finishing write tid 6 to mira09725123-458 2014-08-22 19:30:13,805.805 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 458 v 1877 (ObjNum 824 snap 99 seq_num 824) dirty exists 2014-08-22 19:30:13,811.811 INFO:tasks.rados.rados.0.mira097.stdout:1430: finishing copy_from racing read to mira09725123-133 2014-08-22 19:30:17,158.158 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 1] out_osds: [3, 5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:17,159.159 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:17,159.159 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.753307 2014-08-22 19:30:17,159.159 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.753307199981' 2014-08-22 19:30:17,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.532637 7f660eb1c700 1 librados: starting msgr at :/0 2014-08-22 19:30:17,632.632 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.532649 7f660eb1c700 1 librados: starting objecter 2014-08-22 19:30:17,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.532910 7f660eb1c700 1 librados: setting wanted keys 2014-08-22 19:30:17,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.532918 7f660eb1c700 1 librados: calling monclient init 2014-08-22 19:30:17,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.533766 7f65ff7fe700 10 client.?.objecter ms_handle_connect 0x7f66080355e0 2014-08-22 19:30:17,633.633 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.533798 7f65ff7fe700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:17,672.672 INFO:tasks.rados.rados.0.mira097.stdout:1429: done (3 left) 2014-08-22 19:30:17,672.672 INFO:tasks.rados.rados.0.mira097.stdout:1430: done (2 left) 2014-08-22 19:30:17,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1431: done (1 left) 2014-08-22 19:30:17,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1433: done (0 left) 2014-08-22 19:30:17,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1434: rollback oid 426 current snap is 99 2014-08-22 19:30:17,673.673 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 426 to 84 2014-08-22 19:30:17,673.673 INFO:tasks.rados.rados.0.mira097.stdout:1435: delete oid 111 current snap is 99 2014-08-22 19:30:17,674.674 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.569683 7f660eb1c700 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:17,674.674 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.569805 7f660eb1c700 1 librados: init done 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575759 7f65ff7fe700 10 client.4263.objecter ms_dispatch 0x7f6608000f80 osd_map(254..254 src has 1..254) v3 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575770 7f65ff7fe700 3 client.4263.objecter handle_osd_map got epochs [254,254] > 0 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575773 7f65ff7fe700 3 client.4263.objecter handle_osd_map decoding full epoch 254 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575855 7f65ff7fe700 20 client.4263.objecter dump_active .. 0 homeless 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575881 7f65ff7fe700 10 client.4263.objecter ms_dispatch 0x7f6608000f80 osd_map(254..254 src has 1..254) v3 2014-08-22 19:30:17,675.675 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575883 7f65ff7fe700 3 client.4263.objecter handle_osd_map ignoring epochs [254,254] <= 254 2014-08-22 19:30:17,676.676 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:10.575884 7f65ff7fe700 20 client.4263.objecter dump_active .. 0 homeless 2014-08-22 19:30:17,689.689 INFO:tasks.rados.rados.0.mira097.stdout:1435: done (1 left) 2014-08-22 19:30:17,689.689 INFO:tasks.rados.rados.0.mira097.stdout:1436: snap_remove snap 98 2014-08-22 19:30:17,710.710 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 426 v 2185 (ObjNum 777 snap 72 seq_num 777) dirty exists 2014-08-22 19:30:18,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1434: done (1 left) 2014-08-22 19:30:18,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1436: done (0 left) 2014-08-22 19:30:18,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1437: write oid 23 current snap is 99 2014-08-22 19:30:18,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1437: seq_num 826 ranges {109750=1} 2014-08-22 19:30:18,863.863 INFO:tasks.rados.rados.0.mira097.stdout:1437: writing mira09725123-23 from 109750 to 109751 tid 1 2014-08-22 19:30:18,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1438: read oid 153 snap -1 2014-08-22 19:30:18,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1439: read oid 106 snap -1 2014-08-22 19:30:18,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1440: read oid 244 snap -1 2014-08-22 19:30:18,864.864 INFO:tasks.rados.rados.0.mira097.stdout:1441: delete oid 223 current snap is 99 2014-08-22 19:30:18,875.875 INFO:tasks.rados.rados.0.mira097.stdout:1441: done (4 left) 2014-08-22 19:30:18,876.876 INFO:tasks.rados.rados.0.mira097.stdout:1442: read oid 192 snap 94 2014-08-22 19:30:18,876.876 INFO:tasks.rados.rados.0.mira097.stdout:1443: snap_remove snap 86 2014-08-22 19:30:18,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1437: finishing write tid 1 to mira09725123-23 2014-08-22 19:30:18,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1437: finishing write tid 3 to mira09725123-23 2014-08-22 19:30:18,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1437: finishing write tid 4 to mira09725123-23 2014-08-22 19:30:18,885.885 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 23 v 2195 (ObjNum 826 snap 99 seq_num 826) dirty exists 2014-08-22 19:30:18,889.889 INFO:tasks.rados.rados.0.mira097.stdout:1440: expect (ObjNum 67 snap 0 seq_num 67) 2014-08-22 19:30:18,903.903 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.753307 (8493682) 2014-08-22 19:30:18,906.906 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:11.807297 7f660eb1c700 20 client.4263.objecter shutdown clearing up homeless session... 2014-08-22 19:30:18,907.907 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:11.807732 7f660eb1c700 1 librados: shutdown 2014-08-22 19:30:20,548.548 INFO:tasks.rados.rados.0.mira097.stdout:1438: expect (ObjNum 734 snap 61 seq_num 734) 2014-08-22 19:30:21,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1439: expect (ObjNum 658 snap 46 seq_num 658) 2014-08-22 19:30:22,314.314 INFO:tasks.rados.rados.0.mira097.stdout:1437: done (5 left) 2014-08-22 19:30:22,314.314 INFO:tasks.rados.rados.0.mira097.stdout:1438: done (4 left) 2014-08-22 19:30:22,314.314 INFO:tasks.rados.rados.0.mira097.stdout:1439: done (3 left) 2014-08-22 19:30:22,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1440: done (2 left) 2014-08-22 19:30:22,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1442: done (1 left) 2014-08-22 19:30:22,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1443: done (0 left) 2014-08-22 19:30:22,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1444: copy_from oid 303 from oid 378 current snap is 99 2014-08-22 19:30:22,315.315 INFO:tasks.rados.rados.0.mira097.stdout:1445: copy_from oid 495 from oid 395 current snap is 99 2014-08-22 19:30:22,316.316 INFO:tasks.rados.rados.0.mira097.stdout:1446: rollback oid 221 current snap is 99 2014-08-22 19:30:22,316.316 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 221 to 96 2014-08-22 19:30:22,316.316 INFO:tasks.rados.rados.0.mira097.stdout:1447: snap_remove snap 92 2014-08-22 19:30:22,322.322 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 221 v 1899 (ObjNum 702 snap 52 seq_num 702) dirty exists 2014-08-22 19:30:22,362.362 INFO:tasks.rados.rados.0.mira097.stdout:1445: finishing copy_from to mira09725123-495 2014-08-22 19:30:22,363.363 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 495 v 595 (ObjNum 394 snap 0 seq_num 394) dirty exists 2014-08-22 19:30:22,391.391 INFO:tasks.rados.rados.0.mira097.stdout:1444: finishing copy_from to mira09725123-303 2014-08-22 19:30:22,391.391 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 303 v 466 (ObjNum 377 snap 0 seq_num 377) dirty exists 2014-08-22 19:30:22,486.486 INFO:tasks.rados.rados.0.mira097.stdout:1445: finishing copy_from racing read to mira09725123-495 2014-08-22 19:30:22,486.486 INFO:tasks.rados.rados.0.mira097.stdout:1444: finishing copy_from racing read to mira09725123-303 2014-08-22 19:30:22,788.788 INFO:tasks.rados.rados.0.mira097.stdout:1444: done (3 left) 2014-08-22 19:30:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1445: done (2 left) 2014-08-22 19:30:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1446: done (1 left) 2014-08-22 19:30:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1447: done (0 left) 2014-08-22 19:30:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:1448: rollback oid 140 current snap is 99 2014-08-22 19:30:22,789.789 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 140 to 87 2014-08-22 19:30:22,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1449: delete oid 78 current snap is 99 2014-08-22 19:30:22,843.843 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 140 v 1972 (ObjNum 139 snap 0 seq_num 139) dirty exists 2014-08-22 19:30:22,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1448: done (1 left) 2014-08-22 19:30:22,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1449: done (0 left) 2014-08-22 19:30:22,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1450: copy_from oid 118 from oid 446 current snap is 99 2014-08-22 19:30:22,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1451: copy_from oid 153 from oid 248 current snap is 99 2014-08-22 19:30:22,844.844 INFO:tasks.rados.rados.0.mira097.stdout:1452: snap_remove snap 94 2014-08-22 19:30:22,868.868 INFO:tasks.rados.rados.0.mira097.stdout:1450: finishing copy_from to mira09725123-118 2014-08-22 19:30:22,868.868 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 118 v 1724 (ObjNum 782 snap 73 seq_num 782) dirty exists 2014-08-22 19:30:22,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1450: finishing copy_from racing read to mira09725123-118 2014-08-22 19:30:22,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1451: finishing copy_from to mira09725123-153 2014-08-22 19:30:22,892.892 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 153 v 270 (ObjNum 247 snap 0 seq_num 247) dirty exists 2014-08-22 19:30:22,909.909 INFO:tasks.rados.rados.0.mira097.stdout:1451: finishing copy_from racing read to mira09725123-153 2014-08-22 19:30:23,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1450: done (2 left) 2014-08-22 19:30:23,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1451: done (1 left) 2014-08-22 19:30:23,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1452: done (0 left) 2014-08-22 19:30:23,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1453: rollback oid 477 current snap is 99 2014-08-22 19:30:23,816.816 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 477 to 96 2014-08-22 19:30:23,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1454: read oid 336 snap 90 2014-08-22 19:30:23,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1455: delete oid 409 current snap is 99 2014-08-22 19:30:23,883.883 INFO:tasks.rados.rados.0.mira097.stdout:1455: done (2 left) 2014-08-22 19:30:23,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1456: snap_remove snap 93 2014-08-22 19:30:23,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1454: expect (ObjNum 335 snap 0 seq_num 335) 2014-08-22 19:30:23,910.910 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 1] out_osds: [3, 5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:23,911.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:23,911.911 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2014-08-22 19:30:23,911.911 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:30:23,911.911 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:30:23,912.912 INFO:teuthology.orchestra.run.mira097:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-08-22 19:30:25,567.567 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 477 v 1946 (ObjNum 660 snap 48 seq_num 660) dirty exists 2014-08-22 19:30:25,567.567 INFO:tasks.rados.rados.0.mira097.stdout:1453: done (2 left) 2014-08-22 19:30:25,568.568 INFO:tasks.rados.rados.0.mira097.stdout:1454: done (1 left) 2014-08-22 19:30:25,568.568 INFO:tasks.rados.rados.0.mira097.stdout:1456: done (0 left) 2014-08-22 19:30:25,568.568 INFO:tasks.rados.rados.0.mira097.stdout:1457: write oid 405 current snap is 99 2014-08-22 19:30:25,568.568 INFO:tasks.rados.rados.0.mira097.stdout:1457: seq_num 830 ranges {554797=406020,1579208=559907,2554796=1} 2014-08-22 19:30:25,592.592 INFO:tasks.rados.rados.0.mira097.stdout:1457: writing mira09725123-405 from 554797 to 960817 tid 1 2014-08-22 19:30:25,613.613 INFO:tasks.rados.rados.0.mira097.stdout:1457: writing mira09725123-405 from 1579208 to 2139115 tid 2 2014-08-22 19:30:25,618.618 INFO:tasks.rados.rados.0.mira097.stdout:1457: writing mira09725123-405 from 2554796 to 2554797 tid 3 2014-08-22 19:30:25,619.619 INFO:tasks.rados.rados.0.mira097.stdout:1458: rollback oid 366 current snap is 99 2014-08-22 19:30:25,619.619 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 366 to 87 2014-08-22 19:30:25,619.619 INFO:tasks.rados.rados.0.mira097.stdout:1459: delete oid 302 current snap is 99 2014-08-22 19:30:25,630.630 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 366 v 1950 (ObjNum 365 snap 0 seq_num 365) dirty exists 2014-08-22 19:30:25,652.652 INFO:tasks.rados.rados.0.mira097.stdout:1457: finishing write tid 1 to mira09725123-405 2014-08-22 19:30:25,662.662 INFO:tasks.rados.rados.0.mira097.stdout:1457: finishing write tid 2 to mira09725123-405 2014-08-22 19:30:25,662.662 INFO:tasks.rados.rados.0.mira097.stdout:1457: finishing write tid 3 to mira09725123-405 2014-08-22 19:30:25,739.739 INFO:tasks.rados.rados.0.mira097.stdout:1457: finishing write tid 5 to mira09725123-405 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1457: finishing write tid 6 to mira09725123-405 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 405 v 2227 (ObjNum 830 snap 99 seq_num 830) dirty exists 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1457: done (2 left) 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1458: done (1 left) 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1459: done (0 left) 2014-08-22 19:30:25,740.740 INFO:tasks.rados.rados.0.mira097.stdout:1460: snap_remove snap 88 2014-08-22 19:30:26,190.190 INFO:tasks.rados.rados.0.mira097.stdout:1460: done (0 left) 2014-08-22 19:30:26,190.190 INFO:tasks.rados.rados.0.mira097.stdout:1461: write oid 13 current snap is 99 2014-08-22 19:30:26,190.190 INFO:tasks.rados.rados.0.mira097.stdout:1461: seq_num 832 ranges {594857=502077,1771377=699273,2994856=1} 2014-08-22 19:30:26,217.217 INFO:tasks.rados.rados.0.mira097.stdout:1461: writing mira09725123-13 from 594857 to 1096934 tid 1 2014-08-22 19:30:26,243.243 INFO:tasks.rados.rados.0.mira097.stdout:1461: writing mira09725123-13 from 1771377 to 2470650 tid 2 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout:1461: writing mira09725123-13 from 2994856 to 2994857 tid 3 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout:1462: rollback oid 365 current snap is 99 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 365 to 84 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout:1461: finishing write tid 1 to mira09725123-13 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout:1463: snap_create 2014-08-22 19:30:26,248.248 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:26,363.363 INFO:tasks.rados.rados.0.mira097.stdout:1461: finishing write tid 2 to mira09725123-13 2014-08-22 19:30:26,363.363 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 365 v 2012 (ObjNum 364 snap 0 seq_num 364) dirty exists 2014-08-22 19:30:26,363.363 INFO:tasks.rados.rados.0.mira097.stdout:1462: done (1 left) 2014-08-22 19:30:26,363.363 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:26,363.363 INFO:tasks.rados.rados.0.mira097.stdout:1461: finishing write tid 3 to mira09725123-13 2014-08-22 19:30:26,364.364 INFO:tasks.rados.rados.0.mira097.stdout:1461: finishing write tid 5 to mira09725123-13 2014-08-22 19:30:26,364.364 INFO:tasks.rados.rados.0.mira097.stdout:1461: finishing write tid 6 to mira09725123-13 2014-08-22 19:30:26,364.364 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 13 v 1966 (ObjNum 832 snap 99 seq_num 832) dirty exists 2014-08-22 19:30:26,364.364 INFO:tasks.rados.rados.0.mira097.stdout:1461: done (0 left) 2014-08-22 19:30:27,345.345 INFO:tasks.rados.rados.0.mira097.stdout:1463: done (0 left) 2014-08-22 19:30:27,346.346 INFO:tasks.rados.rados.0.mira097.stdout:1464: read oid 33 snap -1 2014-08-22 19:30:27,346.346 INFO:tasks.rados.rados.0.mira097.stdout:1465: snap_create 2014-08-22 19:30:27,346.346 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:27,352.352 INFO:tasks.rados.rados.0.mira097.stdout:1464: expect (ObjNum 777 snap 72 seq_num 777) 2014-08-22 19:30:27,527.527 INFO:tasks.rados.rados.0.mira097.stdout:1464: done (0 left) 2014-08-22 19:30:28,365.365 INFO:tasks.rados.rados.0.mira097.stdout:1465: done (0 left) 2014-08-22 19:30:28,365.365 INFO:tasks.rados.rados.0.mira097.stdout:1466: write oid 115 current snap is 101 2014-08-22 19:30:28,365.365 INFO:tasks.rados.rados.0.mira097.stdout:1466: seq_num 833 ranges {538663=625305,1884485=769468,3069329=486261,3738662=1} 2014-08-22 19:30:28,391.391 INFO:tasks.rados.rados.0.mira097.stdout:1466: writing mira09725123-115 from 538663 to 1163968 tid 1 2014-08-22 19:30:28,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1466: writing mira09725123-115 from 1884485 to 2653953 tid 2 2014-08-22 19:30:28,437.437 INFO:tasks.rados.rados.0.mira097.stdout:1466: writing mira09725123-115 from 3069329 to 3555590 tid 3 2014-08-22 19:30:28,438.438 INFO:tasks.rados.rados.0.mira097.stdout:1466: writing mira09725123-115 from 3738662 to 3738663 tid 4 2014-08-22 19:30:28,438.438 INFO:tasks.rados.rados.0.mira097.stdout:1467: snap_remove snap 87 2014-08-22 19:30:28,567.567 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 1 to mira09725123-115 2014-08-22 19:30:28,567.567 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 2 to mira09725123-115 2014-08-22 19:30:28,643.643 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 3 to mira09725123-115 2014-08-22 19:30:29,102.102 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 4 to mira09725123-115 2014-08-22 19:30:29,121.121 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 6 to mira09725123-115 2014-08-22 19:30:29,121.121 INFO:tasks.rados.rados.0.mira097.stdout:1466: finishing write tid 7 to mira09725123-115 2014-08-22 19:30:29,121.121 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 115 v 1973 (ObjNum 833 snap 101 seq_num 833) dirty exists 2014-08-22 19:30:29,385.385 INFO:tasks.rados.rados.0.mira097.stdout:1466: done (1 left) 2014-08-22 19:30:29,385.385 INFO:tasks.rados.rados.0.mira097.stdout:1467: done (0 left) 2014-08-22 19:30:29,385.385 INFO:tasks.rados.rados.0.mira097.stdout:1468: copy_from oid 363 from oid 439 current snap is 101 2014-08-22 19:30:29,388.388 INFO:tasks.rados.rados.0.mira097.stdout:1469: rollback oid 280 current snap is 101 2014-08-22 19:30:29,389.389 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 280 to 100 2014-08-22 19:30:29,389.389 INFO:tasks.rados.rados.0.mira097.stdout:1470: write oid 5 current snap is 101 2014-08-22 19:30:29,389.389 INFO:tasks.rados.rados.0.mira097.stdout:1470: seq_num 834 ranges {751268=480757,1715992=439663,2588953=772027,3951267=1} 2014-08-22 19:30:29,414.414 INFO:tasks.rados.rados.0.mira097.stdout:1470: writing mira09725123-5 from 751268 to 1232025 tid 1 2014-08-22 19:30:29,438.438 INFO:tasks.rados.rados.0.mira097.stdout:1470: writing mira09725123-5 from 1715992 to 2155655 tid 2 2014-08-22 19:30:29,474.474 INFO:tasks.rados.rados.0.mira097.stdout:1470: writing mira09725123-5 from 2588953 to 3360980 tid 3 2014-08-22 19:30:29,485.485 INFO:tasks.rados.rados.0.mira097.stdout:1470: writing mira09725123-5 from 3951267 to 3951268 tid 4 2014-08-22 19:30:29,492.492 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 280 v 1976 (ObjNum 279 snap 0 seq_num 279) dirty exists 2014-08-22 19:30:29,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 1 to mira09725123-5 2014-08-22 19:30:29,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1469: done (2 left) 2014-08-22 19:30:29,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1471: read oid 371 snap -1 2014-08-22 19:30:29,493.493 INFO:tasks.rados.rados.0.mira097.stdout:1472: snap_create 2014-08-22 19:30:29,493.493 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:30:29,493.493 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 2 to mira09725123-5 2014-08-22 19:30:29,493.493 INFO:tasks.rados.rados.0.mira097.stdout:1471: done (2 left) 2014-08-22 19:30:29,493.493 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:29,508.508 INFO:tasks.rados.rados.0.mira097.stdout:1468: finishing copy_from to mira09725123-363 2014-08-22 19:30:29,508.508 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 363 v 610 (ObjNum 438 snap 0 seq_num 438) dirty exists 2014-08-22 19:30:29,513.513 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 3 to mira09725123-5 2014-08-22 19:30:29,565.565 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 4 to mira09725123-5 2014-08-22 19:30:29,565.565 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 6 to mira09725123-5 2014-08-22 19:30:29,565.565 INFO:tasks.rados.rados.0.mira097.stdout:1468: finishing copy_from racing read to mira09725123-363 2014-08-22 19:30:29,565.565 INFO:tasks.rados.rados.0.mira097.stdout:1468: done (1 left) 2014-08-22 19:30:29,565.565 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:29,566.566 INFO:tasks.rados.rados.0.mira097.stdout:1470: finishing write tid 7 to mira09725123-5 2014-08-22 19:30:29,566.566 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 5 v 1989 (ObjNum 834 snap 101 seq_num 834) dirty exists 2014-08-22 19:30:29,566.566 INFO:tasks.rados.rados.0.mira097.stdout:1470: done (0 left) 2014-08-22 19:30:30,251.251 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 1] out_osds: [3, 5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:30,251.251 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:30,252.252 INFO:tasks.thrashosds.thrasher:Growing pool rbd 2014-08-22 19:30:30,252.252 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:30:30,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1472: done (0 left) 2014-08-22 19:30:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1473: read oid 293 snap -1 2014-08-22 19:30:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1474: read oid 51 snap -1 2014-08-22 19:30:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1475: read oid 447 snap 84 2014-08-22 19:30:30,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1476: rollback oid 197 current snap is 102 2014-08-22 19:30:30,404.404 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 197 to 84 2014-08-22 19:30:30,404.404 INFO:tasks.rados.rados.0.mira097.stdout:1477: delete oid 363 current snap is 102 2014-08-22 19:30:30,426.426 INFO:tasks.rados.rados.0.mira097.stdout:1474: done (4 left) 2014-08-22 19:30:30,426.426 INFO:tasks.rados.rados.0.mira097.stdout:1477: done (3 left) 2014-08-22 19:30:30,426.426 INFO:tasks.rados.rados.0.mira097.stdout:1478: delete oid 388 current snap is 102 2014-08-22 19:30:30,450.450 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 197 v 2029 (ObjNum 723 snap 58 seq_num 723) dirty exists 2014-08-22 19:30:30,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1476: done (3 left) 2014-08-22 19:30:30,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1478: done (2 left) 2014-08-22 19:30:30,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1479: write oid 195 current snap is 102 2014-08-22 19:30:30,451.451 INFO:tasks.rados.rados.0.mira097.stdout:1479: seq_num 837 ranges {735134=400000} 2014-08-22 19:30:30,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1479: writing mira09725123-195 from 735134 to 1135134 tid 1 2014-08-22 19:30:30,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1480: delete oid 233 current snap is 102 2014-08-22 19:30:30,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1475: expect (ObjNum 446 snap 0 seq_num 446) 2014-08-22 19:30:30,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.602572 7febf0b9b700 1 librados: starting msgr at :/0 2014-08-22 19:30:30,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.602578 7febf0b9b700 1 librados: starting objecter 2014-08-22 19:30:30,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.602728 7febf0b9b700 1 librados: setting wanted keys 2014-08-22 19:30:30,703.703 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.602732 7febf0b9b700 1 librados: calling monclient init 2014-08-22 19:30:30,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.604302 7febe9ae7700 10 client.?.objecter ms_handle_connect 0x7febec0355e0 2014-08-22 19:30:30,704.704 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.604334 7febe9ae7700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:30,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.607416 7febf0b9b700 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:30,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.607518 7febf0b9b700 1 librados: init done 2014-08-22 19:30:30,707.707 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608299 7febe9ae7700 10 client.4270.objecter ms_dispatch 0x7febec000f80 osd_map(264..264 src has 1..264) v3 2014-08-22 19:30:30,708.708 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608309 7febe9ae7700 3 client.4270.objecter handle_osd_map got epochs [264,264] > 0 2014-08-22 19:30:30,708.708 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608312 7febe9ae7700 3 client.4270.objecter handle_osd_map decoding full epoch 264 2014-08-22 19:30:30,708.708 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608450 7febe9ae7700 20 client.4270.objecter dump_active .. 0 homeless 2014-08-22 19:30:30,803.803 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608525 7febe9ae7700 10 client.4270.objecter ms_dispatch 0x7febec000f80 osd_map(264..264 src has 1..264) v3 2014-08-22 19:30:30,803.803 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608531 7febe9ae7700 3 client.4270.objecter handle_osd_map ignoring epochs [264,264] <= 264 2014-08-22 19:30:30,803.803 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.608533 7febe9ae7700 20 client.4270.objecter dump_active .. 0 homeless 2014-08-22 19:30:30,887.887 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:30:31,068.068 INFO:tasks.rados.rados.0.mira097.stdout:1473: expect (ObjNum 505 snap 3 seq_num 505) 2014-08-22 19:30:31,327.327 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.963826 7febf0b9b700 20 client.4270.objecter shutdown clearing up homeless session... 2014-08-22 19:30:31,328.328 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:23.964180 7febf0b9b700 1 librados: shutdown 2014-08-22 19:30:31,517.517 INFO:tasks.ceph.ceph_manager:increase pool size by 10 2014-08-22 19:30:31,518.518 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 94' 2014-08-22 19:30:31,903.903 INFO:tasks.rados.rados.0.mira097.stdout:1479: finishing write tid 1 to mira09725123-195 2014-08-22 19:30:31,903.903 INFO:tasks.rados.rados.0.mira097.stdout:1479: finishing write tid 3 to mira09725123-195 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1479: finishing write tid 4 to mira09725123-195 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 195 v 1988 (ObjNum 837 snap 102 seq_num 837) dirty exists 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1473: done (3 left) 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1475: done (2 left) 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1479: done (1 left) 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1480: done (0 left) 2014-08-22 19:30:31,904.904 INFO:tasks.rados.rados.0.mira097.stdout:1481: copy_from oid 156 from oid 413 current snap is 102 2014-08-22 19:30:31,905.905 INFO:tasks.rados.rados.0.mira097.stdout:1482: write oid 411 current snap is 102 2014-08-22 19:30:31,905.905 INFO:tasks.rados.rados.0.mira097.stdout:1482: seq_num 839 ranges {741145=528323,1913662=577135,3141144=1} 2014-08-22 19:30:31,933.933 INFO:tasks.rados.rados.0.mira097.stdout:1482: writing mira09725123-411 from 741145 to 1269468 tid 1 2014-08-22 19:30:31,959.959 INFO:tasks.rados.rados.0.mira097.stdout:1482: writing mira09725123-411 from 1913662 to 2490797 tid 2 2014-08-22 19:30:31,966.966 INFO:tasks.rados.rados.0.mira097.stdout:1482: writing mira09725123-411 from 3141144 to 3141145 tid 3 2014-08-22 19:30:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:1483: copy_from oid 78 from oid 104 current snap is 102 2014-08-22 19:30:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:1484: snap_create 2014-08-22 19:30:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:30:31,967.967 INFO:tasks.rados.rados.0.mira097.stdout:1482: finishing write tid 1 to mira09725123-411 2014-08-22 19:30:31,969.969 INFO:tasks.rados.rados.0.mira097.stdout:1482: finishing write tid 2 to mira09725123-411 2014-08-22 19:30:31,972.972 INFO:tasks.rados.rados.0.mira097.stdout:1482: finishing write tid 3 to mira09725123-411 2014-08-22 19:30:31,972.972 INFO:tasks.rados.rados.0.mira097.stdout:1482: finishing write tid 5 to mira09725123-411 2014-08-22 19:30:31,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1482: finishing write tid 6 to mira09725123-411 2014-08-22 19:30:31,978.978 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 411 v 2011 (ObjNum 839 snap 102 seq_num 839) dirty exists 2014-08-22 19:30:31,978.978 INFO:tasks.rados.rados.0.mira097.stdout:1482: done (2 left) 2014-08-22 19:30:31,978.978 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:32,026.026 INFO:tasks.rados.rados.0.mira097.stdout:1481: finishing copy_from to mira09725123-156 2014-08-22 19:30:32,027.027 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 156 v 596 (ObjNum 487 snap 0 seq_num 487) dirty exists 2014-08-22 19:30:32,037.037 INFO:tasks.rados.rados.0.mira097.stdout:1481: finishing copy_from racing read to mira09725123-156 2014-08-22 19:30:32,037.037 INFO:tasks.rados.rados.0.mira097.stdout:1481: done (1 left) 2014-08-22 19:30:32,037.037 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:32,147.147 INFO:tasks.rados.rados.0.mira097.stdout:1483: finishing copy_from to mira09725123-78 2014-08-22 19:30:32,147.147 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 78 v 124 (ObjNum 103 snap 0 seq_num 103) dirty exists 2014-08-22 19:30:32,147.147 INFO:tasks.rados.rados.0.mira097.stdout:1483: finishing copy_from racing read to mira09725123-78 2014-08-22 19:30:32,148.148 INFO:tasks.rados.rados.0.mira097.stdout:1483: done (0 left) 2014-08-22 19:30:32,184.184 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.084724 7f5066510700 1 librados: starting msgr at :/0 2014-08-22 19:30:32,185.185 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.084734 7f5066510700 1 librados: starting objecter 2014-08-22 19:30:32,185.185 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.084971 7f5066510700 1 librados: setting wanted keys 2014-08-22 19:30:32,185.185 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.084980 7f5066510700 1 librados: calling monclient init 2014-08-22 19:30:32,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.086589 7f505effd700 10 client.?.objecter ms_handle_connect 0x7f50600355e0 2014-08-22 19:30:32,186.186 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.086613 7f505effd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:32,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.088606 7f5066510700 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:32,187.187 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.088721 7f5066510700 1 librados: init done 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089064 7f505effd700 10 client.4266.objecter ms_dispatch 0x7f5060000f80 osd_map(264..264 src has 1..264) v3 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089074 7f505effd700 3 client.4266.objecter handle_osd_map got epochs [264,264] > 0 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089078 7f505effd700 3 client.4266.objecter handle_osd_map decoding full epoch 264 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089174 7f505effd700 20 client.4266.objecter dump_active .. 0 homeless 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089235 7f505effd700 10 client.4266.objecter ms_dispatch 0x7f5060000f80 osd_map(264..264 src has 1..264) v3 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089245 7f505effd700 3 client.4266.objecter handle_osd_map ignoring epochs [264,264] <= 264 2014-08-22 19:30:32,189.189 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.089248 7f505effd700 20 client.4266.objecter dump_active .. 0 homeless 2014-08-22 19:30:32,420.420 INFO:tasks.rados.rados.0.mira097.stdout:1484: done (0 left) 2014-08-22 19:30:32,420.420 INFO:tasks.rados.rados.0.mira097.stdout:1485: copy_from oid 244 from oid 336 current snap is 103 2014-08-22 19:30:32,421.421 INFO:tasks.rados.rados.0.mira097.stdout:1486: copy_from oid 7 from oid 465 current snap is 103 2014-08-22 19:30:32,421.421 INFO:tasks.rados.rados.0.mira097.stdout:1487: rollback oid 211 current snap is 103 2014-08-22 19:30:32,421.421 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 211 to 102 2014-08-22 19:30:32,421.421 INFO:tasks.rados.rados.0.mira097.stdout:1488: snap_remove snap 99 2014-08-22 19:30:32,433.433 INFO:teuthology.orchestra.run.mira097.stderr:set pool 0 pg_num to 94 2014-08-22 19:30:32,445.445 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.345068 7f5066510700 20 client.4266.objecter shutdown clearing up homeless session... 2014-08-22 19:30:32,445.445 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:25.345921 7f5066510700 1 librados: shutdown 2014-08-22 19:30:32,500.500 INFO:tasks.rados.rados.0.mira097.stdout:1486: finishing copy_from to mira09725123-7 2014-08-22 19:30:32,500.500 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 7 v 643 (ObjNum 464 snap 0 seq_num 464) dirty exists 2014-08-22 19:30:32,500.500 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 211 v 2249 (ObjNum 503 snap 3 seq_num 503) dirty exists 2014-08-22 19:30:32,534.534 INFO:tasks.rados.rados.0.mira097.stdout:1485: finishing copy_from to mira09725123-244 2014-08-22 19:30:32,535.535 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 244 v 425 (ObjNum 335 snap 0 seq_num 335) dirty exists 2014-08-22 19:30:32,545.545 INFO:tasks.rados.rados.0.mira097.stdout:1486: finishing copy_from racing read to mira09725123-7 2014-08-22 19:30:32,553.553 INFO:tasks.rados.rados.0.mira097.stdout:1485: finishing copy_from racing read to mira09725123-244 2014-08-22 19:30:33,423.423 INFO:tasks.rados.rados.0.mira097.stdout:1485: done (3 left) 2014-08-22 19:30:33,423.423 INFO:tasks.rados.rados.0.mira097.stdout:1486: done (2 left) 2014-08-22 19:30:33,423.423 INFO:tasks.rados.rados.0.mira097.stdout:1487: done (1 left) 2014-08-22 19:30:33,424.424 INFO:tasks.rados.rados.0.mira097.stdout:1488: done (0 left) 2014-08-22 19:30:33,424.424 INFO:tasks.rados.rados.0.mira097.stdout:1489: read oid 141 snap -1 2014-08-22 19:30:33,424.424 INFO:tasks.rados.rados.0.mira097.stdout:1490: snap_create 2014-08-22 19:30:33,424.424 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:33,448.448 INFO:tasks.rados.rados.0.mira097.stdout:1489: expect (ObjNum 754 snap 67 seq_num 754) 2014-08-22 19:30:33,745.745 INFO:tasks.rados.rados.0.mira097.stdout:1489: done (0 left) 2014-08-22 19:30:34,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1490: done (0 left) 2014-08-22 19:30:34,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1491: snap_remove snap 101 2014-08-22 19:30:35,447.447 INFO:tasks.rados.rados.0.mira097.stdout:1491: done (0 left) 2014-08-22 19:30:35,448.448 INFO:tasks.rados.rados.0.mira097.stdout:1492: write oid 336 current snap is 104 2014-08-22 19:30:35,448.448 INFO:tasks.rados.rados.0.mira097.stdout:1492: seq_num 840 ranges {436375=500127,1625746=798754,2836374=1} 2014-08-22 19:30:35,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1492: writing mira09725123-336 from 436375 to 936502 tid 1 2014-08-22 19:30:35,500.500 INFO:tasks.rados.rados.0.mira097.stdout:1492: writing mira09725123-336 from 1625746 to 2424500 tid 2 2014-08-22 19:30:35,504.504 INFO:tasks.rados.rados.0.mira097.stdout:1492: writing mira09725123-336 from 2836374 to 2836375 tid 3 2014-08-22 19:30:35,505.505 INFO:tasks.rados.rados.0.mira097.stdout:1493: write oid 404 current snap is 104 2014-08-22 19:30:35,505.505 INFO:tasks.rados.rados.0.mira097.stdout:1493: seq_num 841 ranges {531605=623356,1855207=676398} 2014-08-22 19:30:35,529.529 INFO:tasks.rados.rados.0.mira097.stdout:1493: writing mira09725123-404 from 531605 to 1154961 tid 1 2014-08-22 19:30:35,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1493: writing mira09725123-404 from 1855207 to 2531605 tid 2 2014-08-22 19:30:35,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1494: delete oid 14 current snap is 104 2014-08-22 19:30:35,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1492: finishing write tid 1 to mira09725123-336 2014-08-22 19:30:35,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1492: finishing write tid 2 to mira09725123-336 2014-08-22 19:30:35,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1492: finishing write tid 3 to mira09725123-336 2014-08-22 19:30:35,559.559 INFO:tasks.rados.rados.0.mira097.stdout:1492: finishing write tid 5 to mira09725123-336 2014-08-22 19:30:35,621.621 INFO:tasks.rados.rados.0.mira097.stdout:1492: finishing write tid 6 to mira09725123-336 2014-08-22 19:30:35,621.621 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 336 v 2025 (ObjNum 840 snap 104 seq_num 840) dirty exists 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1493: finishing write tid 1 to mira09725123-404 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1493: finishing write tid 2 to mira09725123-404 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1493: finishing write tid 4 to mira09725123-404 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1493: finishing write tid 5 to mira09725123-404 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 404 v 2029 (ObjNum 841 snap 104 seq_num 841) dirty exists 2014-08-22 19:30:35,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1492: done (2 left) 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1493: done (1 left) 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1494: done (0 left) 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1495: rollback oid 160 current snap is 104 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 160 to 100 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1496: snap_remove snap 103 2014-08-22 19:30:35,623.623 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 160 v 2028 (ObjNum 159 snap 0 seq_num 159) dirty exists 2014-08-22 19:30:36,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1495: done (1 left) 2014-08-22 19:30:36,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1496: done (0 left) 2014-08-22 19:30:36,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1497: read oid 263 snap -1 2014-08-22 19:30:36,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1498: snap_create 2014-08-22 19:30:36,459.459 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:36,555.555 INFO:tasks.rados.rados.0.mira097.stdout:1497: expect (ObjNum 719 snap 56 seq_num 719) 2014-08-22 19:30:37,457.457 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4, 1] out_osds: [3, 5] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:37,457.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:37,457.457 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [2, 0, 4, 1] 2014-08-22 19:30:37,458.458 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 1' 2014-08-22 19:30:37,907.907 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.805296 7f0e9523e700 1 librados: starting msgr at :/0 2014-08-22 19:30:37,908.908 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.805302 7f0e9523e700 1 librados: starting objecter 2014-08-22 19:30:37,908.908 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.805450 7f0e9523e700 1 librados: setting wanted keys 2014-08-22 19:30:37,908.908 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.805457 7f0e9523e700 1 librados: calling monclient init 2014-08-22 19:30:37,908.908 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.806175 7f0e8e2e8700 10 client.?.objecter ms_handle_connect 0x7f0e900355e0 2014-08-22 19:30:37,908.908 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.806208 7f0e8e2e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:37,912.912 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.807725 7f0e9523e700 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:37,913.913 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.807815 7f0e9523e700 1 librados: init done 2014-08-22 19:30:37,913.913 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808120 7f0e8e2e8700 10 client.4269.objecter ms_dispatch 0x7f0e90000f80 osd_map(269..269 src has 1..269) v3 2014-08-22 19:30:37,913.913 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808131 7f0e8e2e8700 3 client.4269.objecter handle_osd_map got epochs [269,269] > 0 2014-08-22 19:30:37,913.913 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808134 7f0e8e2e8700 3 client.4269.objecter handle_osd_map decoding full epoch 269 2014-08-22 19:30:37,914.914 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808214 7f0e8e2e8700 20 client.4269.objecter dump_active .. 0 homeless 2014-08-22 19:30:37,914.914 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808256 7f0e8e2e8700 10 client.4269.objecter ms_dispatch 0x7f0e90000f80 osd_map(269..269 src has 1..269) v3 2014-08-22 19:30:37,914.914 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808260 7f0e8e2e8700 3 client.4269.objecter handle_osd_map ignoring epochs [269,269] <= 269 2014-08-22 19:30:38,006.006 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:30.808261 7f0e8e2e8700 20 client.4269.objecter dump_active .. 0 homeless 2014-08-22 19:30:38,370.370 INFO:tasks.rados.rados.0.mira097.stdout:1497: done (0 left) 2014-08-22 19:30:38,464.464 INFO:tasks.rados.rados.0.mira097.stdout:1498: done (0 left) 2014-08-22 19:30:38,464.464 INFO:tasks.rados.rados.0.mira097.stdout:1499: copy_from oid 230 from oid 371 current snap is 105 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1500: read oid 159 snap -1 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1501: read oid 456 snap -1 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1502: read oid 74 snap 96 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1503: write oid 26 current snap is 105 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1503: seq_num 843 ranges {536591=1} 2014-08-22 19:30:38,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1503: writing mira09725123-26 from 536591 to 536592 tid 1 2014-08-22 19:30:38,466.466 INFO:tasks.rados.rados.0.mira097.stdout:1504: delete oid 454 current snap is 105 2014-08-22 19:30:38,479.479 INFO:tasks.rados.rados.0.mira097.stdout:1502: expect (ObjNum 213 snap 0 seq_num 213) 2014-08-22 19:30:38,488.488 INFO:teuthology.orchestra.run.mira097.stderr:marked out osd.1. 2014-08-22 19:30:38,491.491 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:31.391488 7f0e9523e700 20 client.4269.objecter shutdown clearing up homeless session... 2014-08-22 19:30:38,668.668 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:31.391846 7f0e9523e700 1 librados: shutdown 2014-08-22 19:30:38,668.668 INFO:tasks.rados.rados.0.mira097.stdout:1501: expect (ObjNum 615 snap 29 seq_num 615) 2014-08-22 19:30:39,662.662 INFO:tasks.rados.rados.0.mira097.stdout:1503: finishing write tid 1 to mira09725123-26 2014-08-22 19:30:39,665.665 INFO:tasks.rados.rados.0.mira097.stdout:1503: finishing write tid 3 to mira09725123-26 2014-08-22 19:30:39,669.669 INFO:tasks.rados.rados.0.mira097.stdout:1503: finishing write tid 4 to mira09725123-26 2014-08-22 19:30:39,669.669 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 26 v 2034 (ObjNum 843 snap 105 seq_num 843) dirty exists 2014-08-22 19:30:39,676.676 INFO:tasks.rados.rados.0.mira097.stdout:1499: finishing copy_from to mira09725123-230 2014-08-22 19:30:39,676.676 INFO:tasks.rados.rados.0.mira097.stdout:1499: got expected ENOENT (src dne) 2014-08-22 19:30:39,676.676 INFO:tasks.rados.rados.0.mira097.stdout:1499: finishing copy_from racing read to mira09725123-230 2014-08-22 19:30:39,707.707 INFO:tasks.rados.rados.0.mira097.stdout:1500: expect (ObjNum 158 snap 0 seq_num 158) 2014-08-22 19:30:39,891.891 INFO:tasks.rados.rados.0.mira097.stdout:1499: done (5 left) 2014-08-22 19:30:39,891.891 INFO:tasks.rados.rados.0.mira097.stdout:1500: done (4 left) 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1501: done (3 left) 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1502: done (2 left) 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1503: done (1 left) 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1504: done (0 left) 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1505: read oid 78 snap -1 2014-08-22 19:30:39,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1506: delete oid 41 current snap is 105 2014-08-22 19:30:39,895.895 INFO:tasks.rados.rados.0.mira097.stdout:1505: expect (ObjNum 103 snap 0 seq_num 103) 2014-08-22 19:30:40,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1505: done (1 left) 2014-08-22 19:30:40,431.431 INFO:tasks.rados.rados.0.mira097.stdout:1506: done (0 left) 2014-08-22 19:30:40,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1507: write oid 323 current snap is 105 2014-08-22 19:30:40,432.432 INFO:tasks.rados.rados.0.mira097.stdout:1507: seq_num 846 ranges {521481=554569,1687946=33535} 2014-08-22 19:30:40,458.458 INFO:tasks.rados.rados.0.mira097.stdout:1507: writing mira09725123-323 from 521481 to 1076050 tid 1 2014-08-22 19:30:40,464.464 INFO:tasks.rados.rados.0.mira097.stdout:1507: writing mira09725123-323 from 1687946 to 1721481 tid 2 2014-08-22 19:30:40,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1508: read oid 314 snap 97 2014-08-22 19:30:40,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1509: copy_from oid 118 from oid 57 current snap is 105 2014-08-22 19:30:40,465.465 INFO:tasks.rados.rados.0.mira097.stdout:1510: snap_create 2014-08-22 19:30:40,466.466 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:30:40,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1507: finishing write tid 1 to mira09725123-323 2014-08-22 19:30:40,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1507: finishing write tid 2 to mira09725123-323 2014-08-22 19:30:40,503.503 INFO:tasks.rados.rados.0.mira097.stdout:1507: finishing write tid 4 to mira09725123-323 2014-08-22 19:30:40,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1507: finishing write tid 5 to mira09725123-323 2014-08-22 19:30:40,603.603 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 323 v 2276 (ObjNum 846 snap 105 seq_num 846) dirty exists 2014-08-22 19:30:40,603.603 INFO:tasks.rados.rados.0.mira097.stdout:1507: done (2 left) 2014-08-22 19:30:40,603.603 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:40,604.604 INFO:tasks.rados.rados.0.mira097.stdout:1508: expect (ObjNum 313 snap 0 seq_num 313) 2014-08-22 19:30:40,755.755 INFO:tasks.rados.rados.0.mira097.stdout:1509: finishing copy_from to mira09725123-118 2014-08-22 19:30:40,755.755 INFO:tasks.rados.rados.0.mira097.stdout:1509: got expected ENOENT (src dne) 2014-08-22 19:30:40,755.755 INFO:tasks.rados.rados.0.mira097.stdout:1509: finishing copy_from racing read to mira09725123-118 2014-08-22 19:30:40,756.756 INFO:tasks.rados.rados.0.mira097.stdout:1508: done (1 left) 2014-08-22 19:30:40,756.756 INFO:tasks.rados.rados.0.mira097.stdout:1509: done (0 left) 2014-08-22 19:30:41,470.470 INFO:tasks.rados.rados.0.mira097.stdout:1510: done (0 left) 2014-08-22 19:30:41,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1511: copy_from oid 255 from oid 194 current snap is 106 2014-08-22 19:30:41,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1512: read oid 65 snap -1 2014-08-22 19:30:41,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1513: write oid 38 current snap is 106 2014-08-22 19:30:41,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1513: seq_num 847 ranges {465287=677798,1649631=676617,2465286=1} 2014-08-22 19:30:41,502.502 INFO:tasks.rados.rados.0.mira097.stdout:1513: writing mira09725123-38 from 465287 to 1143085 tid 1 2014-08-22 19:30:41,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1513: writing mira09725123-38 from 1649631 to 2326248 tid 2 2014-08-22 19:30:41,528.528 INFO:tasks.rados.rados.0.mira097.stdout:1513: writing mira09725123-38 from 2465286 to 2465287 tid 3 2014-08-22 19:30:41,529.529 INFO:tasks.rados.rados.0.mira097.stdout:1514: snap_create 2014-08-22 19:30:41,529.529 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:30:41,537.537 INFO:tasks.rados.rados.0.mira097.stdout:1513: finishing write tid 1 to mira09725123-38 2014-08-22 19:30:41,546.546 INFO:tasks.rados.rados.0.mira097.stdout:1513: finishing write tid 2 to mira09725123-38 2014-08-22 19:30:41,546.546 INFO:tasks.rados.rados.0.mira097.stdout:1513: finishing write tid 3 to mira09725123-38 2014-08-22 19:30:41,547.547 INFO:tasks.rados.rados.0.mira097.stdout:1513: finishing write tid 5 to mira09725123-38 2014-08-22 19:30:41,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1513: finishing write tid 6 to mira09725123-38 2014-08-22 19:30:41,642.642 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 38 v 2042 (ObjNum 847 snap 106 seq_num 847) dirty exists 2014-08-22 19:30:41,642.642 INFO:tasks.rados.rados.0.mira097.stdout:1513: done (2 left) 2014-08-22 19:30:41,643.643 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:42,831.831 INFO:tasks.rados.rados.0.mira097.stdout:1511: finishing copy_from to mira09725123-255 2014-08-22 19:30:42,831.831 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 255 v 1682 (ObjNum 740 snap 63 seq_num 740) dirty exists 2014-08-22 19:30:42,854.854 INFO:tasks.rados.rados.0.mira097.stdout:1512: expect (ObjNum 64 snap 0 seq_num 64) 2014-08-22 19:30:43,557.557 INFO:tasks.rados.rados.0.mira097.stdout:1511: finishing copy_from racing read to mira09725123-255 2014-08-22 19:30:43,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1511: done (1 left) 2014-08-22 19:30:43,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1512: done (0 left) 2014-08-22 19:30:43,644.644 INFO:tasks.rados.rados.0.mira097.stdout:1514: done (0 left) 2014-08-22 19:30:43,645.645 INFO:tasks.rados.rados.0.mira097.stdout:1515: copy_from oid 472 from oid 182 current snap is 107 2014-08-22 19:30:43,645.645 INFO:tasks.rados.rados.0.mira097.stdout:1516: write oid 205 current snap is 107 2014-08-22 19:30:43,645.645 INFO:tasks.rados.rados.0.mira097.stdout:1516: seq_num 848 ranges {677893=498178,1727667=746812,2677892=1} 2014-08-22 19:30:43,667.667 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4] out_osds: [3, 5, 1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:43,667.667 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:43,667.667 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2014-08-22 19:30:43,668.668 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 2 0' 2014-08-22 19:30:43,668.668 INFO:tasks.rados.rados.0.mira097.stdout:1516: writing mira09725123-205 from 677893 to 1176071 tid 1 2014-08-22 19:30:43,693.693 INFO:tasks.rados.rados.0.mira097.stdout:1516: writing mira09725123-205 from 1727667 to 2474479 tid 2 2014-08-22 19:30:43,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1516: writing mira09725123-205 from 2677892 to 2677893 tid 3 2014-08-22 19:30:43,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1517: read oid 154 snap -1 2014-08-22 19:30:43,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1518: copy_from oid 206 from oid 351 current snap is 107 2014-08-22 19:30:43,730.730 INFO:tasks.rados.rados.0.mira097.stdout:1519: snap_create 2014-08-22 19:30:43,731.731 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:30:43,731.731 INFO:tasks.rados.rados.0.mira097.stdout:1517: done (3 left) 2014-08-22 19:30:43,731.731 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:30:43,731.731 INFO:tasks.rados.rados.0.mira097.stdout:1518: finishing copy_from to mira09725123-206 2014-08-22 19:30:43,731.731 INFO:tasks.rados.rados.0.mira097.stdout:1518: got expected ENOENT (src dne) 2014-08-22 19:30:44,122.122 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.023208 7f940fed2700 1 librados: starting msgr at :/0 2014-08-22 19:30:44,123.123 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.023214 7f940fed2700 1 librados: starting objecter 2014-08-22 19:30:44,303.303 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.023311 7f940fed2700 1 librados: setting wanted keys 2014-08-22 19:30:44,303.303 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.023320 7f940fed2700 1 librados: calling monclient init 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.024980 7f940cf7f700 10 client.?.objecter ms_handle_connect 0x7f94080355e0 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.025000 7f940cf7f700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.028503 7f940fed2700 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.028581 7f940fed2700 1 librados: init done 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029527 7f940cf7f700 10 client.4273.objecter ms_dispatch 0x7f9408000f80 osd_map(273..273 src has 1..273) v3 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029533 7f940cf7f700 3 client.4273.objecter handle_osd_map got epochs [273,273] > 0 2014-08-22 19:30:44,304.304 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029535 7f940cf7f700 3 client.4273.objecter handle_osd_map decoding full epoch 273 2014-08-22 19:30:44,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029612 7f940cf7f700 20 client.4273.objecter dump_active .. 0 homeless 2014-08-22 19:30:44,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029650 7f940cf7f700 10 client.4273.objecter ms_dispatch 0x7f9408000f80 osd_map(273..273 src has 1..273) v3 2014-08-22 19:30:44,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029652 7f940cf7f700 3 client.4273.objecter handle_osd_map ignoring epochs [273,273] <= 273 2014-08-22 19:30:44,305.305 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:37.029654 7f940cf7f700 20 client.4273.objecter dump_active .. 0 homeless 2014-08-22 19:30:44,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1518: finishing copy_from racing read to mira09725123-206 2014-08-22 19:30:44,433.433 INFO:tasks.rados.rados.0.mira097.stdout:1518: done (2 left) 2014-08-22 19:30:44,433.433 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:30:44,441.441 INFO:tasks.rados.rados.0.mira097.stdout:1515: finishing copy_from to mira09725123-472 2014-08-22 19:30:44,441.441 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 472 v 27 (ObjNum 22 snap 0 seq_num 22) dirty exists 2014-08-22 19:30:44,442.442 INFO:tasks.rados.rados.0.mira097.stdout:1515: finishing copy_from racing read to mira09725123-472 2014-08-22 19:30:44,442.442 INFO:tasks.rados.rados.0.mira097.stdout:1515: done (1 left) 2014-08-22 19:30:44,442.442 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:44,487.487 INFO:tasks.rados.rados.0.mira097.stdout:1516: finishing write tid 1 to mira09725123-205 2014-08-22 19:30:44,487.487 INFO:tasks.rados.rados.0.mira097.stdout:1516: finishing write tid 2 to mira09725123-205 2014-08-22 19:30:44,488.488 INFO:tasks.rados.rados.0.mira097.stdout:1516: finishing write tid 3 to mira09725123-205 2014-08-22 19:30:44,488.488 INFO:tasks.rados.rados.0.mira097.stdout:1516: finishing write tid 5 to mira09725123-205 2014-08-22 19:30:44,605.605 INFO:tasks.rados.rados.0.mira097.stdout:1516: finishing write tid 6 to mira09725123-205 2014-08-22 19:30:44,605.605 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 205 v 2057 (ObjNum 848 snap 107 seq_num 848) dirty exists 2014-08-22 19:30:44,605.605 INFO:tasks.rados.rados.0.mira097.stdout:1516: done (0 left) 2014-08-22 19:30:46,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1519: done (0 left) 2014-08-22 19:30:46,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1520: write oid 279 current snap is 108 2014-08-22 19:30:46,197.197 INFO:tasks.rados.rados.0.mira097.stdout:1520: seq_num 849 ranges {621698=623454,1807750=568431,2832038=589660} 2014-08-22 19:30:46,211.211 INFO:teuthology.orchestra.run.mira097.stderr:set osd.2 primary-affinity to 0 (802) 2014-08-22 19:30:46,215.215 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:39.116091 7f940fed2700 20 client.4273.objecter shutdown clearing up homeless session... 2014-08-22 19:30:46,216.216 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:39.116448 7f940fed2700 1 librados: shutdown 2014-08-22 19:30:46,226.226 INFO:tasks.rados.rados.0.mira097.stdout:1520: writing mira09725123-279 from 621698 to 1245152 tid 1 2014-08-22 19:30:46,247.247 INFO:tasks.rados.rados.0.mira097.stdout:1520: writing mira09725123-279 from 1807750 to 2376181 tid 2 2014-08-22 19:30:46,268.268 INFO:tasks.rados.rados.0.mira097.stdout:1520: writing mira09725123-279 from 2832038 to 3421698 tid 3 2014-08-22 19:30:46,268.268 INFO:tasks.rados.rados.0.mira097.stdout:1521: snap_create 2014-08-22 19:30:46,269.269 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:30:46,269.269 INFO:tasks.rados.rados.0.mira097.stdout:1520: finishing write tid 1 to mira09725123-279 2014-08-22 19:30:46,269.269 INFO:tasks.rados.rados.0.mira097.stdout:1520: finishing write tid 2 to mira09725123-279 2014-08-22 19:30:46,279.279 INFO:tasks.rados.rados.0.mira097.stdout:1520: finishing write tid 3 to mira09725123-279 2014-08-22 19:30:46,372.372 INFO:tasks.rados.rados.0.mira097.stdout:1520: finishing write tid 5 to mira09725123-279 2014-08-22 19:30:46,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1520: finishing write tid 6 to mira09725123-279 2014-08-22 19:30:46,373.373 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 279 v 2062 (ObjNum 849 snap 108 seq_num 849) dirty exists 2014-08-22 19:30:46,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1520: done (0 left) 2014-08-22 19:30:49,102.102 INFO:tasks.rados.rados.0.mira097.stdout:1521: done (0 left) 2014-08-22 19:30:49,102.102 INFO:tasks.rados.rados.0.mira097.stdout:1522: read oid 419 snap 90 2014-08-22 19:30:49,102.102 INFO:tasks.rados.rados.0.mira097.stdout:1523: snap_remove snap 105 2014-08-22 19:30:50,114.114 INFO:tasks.rados.rados.0.mira097.stdout:1522: done (1 left) 2014-08-22 19:30:50,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1523: done (0 left) 2014-08-22 19:30:50,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1524: write oid 494 current snap is 109 2014-08-22 19:30:50,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1524: seq_num 850 ranges {165503=1} 2014-08-22 19:30:50,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1524: writing mira09725123-494 from 165503 to 165504 tid 1 2014-08-22 19:30:50,115.115 INFO:tasks.rados.rados.0.mira097.stdout:1525: rollback oid 466 current snap is 109 2014-08-22 19:30:50,116.116 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 466 to 106 2014-08-22 19:30:50,116.116 INFO:tasks.rados.rados.0.mira097.stdout:1526: delete oid 36 current snap is 109 2014-08-22 19:30:50,132.132 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 466 v 0 (ObjNum 1 snap 1 seq_num 37180752) dirty dne 2014-08-22 19:30:50,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1525: done (2 left) 2014-08-22 19:30:50,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1526: done (1 left) 2014-08-22 19:30:50,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1527: read oid 475 snap -1 2014-08-22 19:30:50,152.152 INFO:tasks.rados.rados.0.mira097.stdout:1528: read oid 8 snap -1 2014-08-22 19:30:50,153.153 INFO:tasks.rados.rados.0.mira097.stdout:1529: write oid 270 current snap is 109 2014-08-22 19:30:50,153.153 INFO:tasks.rados.rados.0.mira097.stdout:1529: seq_num 852 ranges {605563=1} 2014-08-22 19:30:50,157.157 INFO:tasks.rados.rados.0.mira097.stdout:1529: writing mira09725123-270 from 605563 to 605564 tid 1 2014-08-22 19:30:50,157.157 INFO:tasks.rados.rados.0.mira097.stdout:1530: read oid 332 snap 106 2014-08-22 19:30:50,287.287 INFO:tasks.rados.rados.0.mira097.stdout:1524: finishing write tid 1 to mira09725123-494 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1524: finishing write tid 3 to mira09725123-494 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1524: finishing write tid 4 to mira09725123-494 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 494 v 2057 (ObjNum 850 snap 109 seq_num 850) dirty exists 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1524: done (4 left) 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1531: write oid 365 current snap is 109 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1531: seq_num 853 ranges {666745=698192,1866744=1} 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1531: writing mira09725123-365 from 666745 to 1364937 tid 1 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1531: writing mira09725123-365 from 1866744 to 1866745 tid 2 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1532: delete oid 244 current snap is 109 2014-08-22 19:30:50,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1527: expect (ObjNum 474 snap 0 seq_num 474) 2014-08-22 19:30:51,221.221 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4] out_osds: [3, 5, 1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:51,221.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:51,221.221 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.952317 2014-08-22 19:30:51,222.222 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.952317038202' 2014-08-22 19:30:51,669.669 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.569070 7f00d5afe700 1 librados: starting msgr at :/0 2014-08-22 19:30:51,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.569075 7f00d5afe700 1 librados: starting objecter 2014-08-22 19:30:51,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.569215 7f00d5afe700 1 librados: setting wanted keys 2014-08-22 19:30:51,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.569220 7f00d5afe700 1 librados: calling monclient init 2014-08-22 19:30:51,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.569993 7f00ce7fc700 10 client.?.objecter ms_handle_connect 0x7f00d00355e0 2014-08-22 19:30:51,670.670 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.570015 7f00ce7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571347 7f00d5afe700 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571450 7f00d5afe700 1 librados: init done 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571744 7f00ce7fc700 10 client.4256.objecter ms_dispatch 0x7f00d0000f80 osd_map(276..276 src has 1..276) v3 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571753 7f00ce7fc700 3 client.4256.objecter handle_osd_map got epochs [276,276] > 0 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571754 7f00ce7fc700 3 client.4256.objecter handle_osd_map decoding full epoch 276 2014-08-22 19:30:51,672.672 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571843 7f00ce7fc700 20 client.4256.objecter dump_active .. 0 homeless 2014-08-22 19:30:51,673.673 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571876 7f00ce7fc700 10 client.4256.objecter ms_dispatch 0x7f00d0000f80 osd_map(276..276 src has 1..276) v3 2014-08-22 19:30:51,673.673 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571880 7f00ce7fc700 3 client.4256.objecter handle_osd_map ignoring epochs [276,276] <= 276 2014-08-22 19:30:51,673.673 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:44.571882 7f00ce7fc700 20 client.4256.objecter dump_active .. 0 homeless 2014-08-22 19:30:51,897.897 INFO:tasks.rados.rados.0.mira097.stdout:1528: expect (ObjNum 162 snap 0 seq_num 162) 2014-08-22 19:30:52,811.811 INFO:tasks.rados.rados.0.mira097.stdout:1530: expect (ObjNum 743 snap 64 seq_num 743) 2014-08-22 19:30:53,673.673 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.952317 (8624112) 2014-08-22 19:30:53,678.678 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:46.567321 7f00d5afe700 20 client.4256.objecter shutdown clearing up homeless session... 2014-08-22 19:30:53,678.678 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:46.567710 7f00d5afe700 1 librados: shutdown 2014-08-22 19:30:53,942.942 INFO:tasks.rados.rados.0.mira097.stdout:1529: finishing write tid 1 to mira09725123-270 2014-08-22 19:30:53,942.942 INFO:tasks.rados.rados.0.mira097.stdout:1529: finishing write tid 3 to mira09725123-270 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1529: finishing write tid 4 to mira09725123-270 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 270 v 2071 (ObjNum 852 snap 109 seq_num 852) dirty exists 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1531: finishing write tid 1 to mira09725123-365 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1531: finishing write tid 2 to mira09725123-365 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1531: finishing write tid 4 to mira09725123-365 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1531: finishing write tid 5 to mira09725123-365 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 365 v 2079 (ObjNum 853 snap 109 seq_num 853) dirty exists 2014-08-22 19:30:53,943.943 INFO:tasks.rados.rados.0.mira097.stdout:1527: done (5 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1528: done (4 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1529: done (3 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1530: done (2 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1531: done (1 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1532: done (0 left) 2014-08-22 19:30:53,944.944 INFO:tasks.rados.rados.0.mira097.stdout:1533: snap_create 2014-08-22 19:30:54,669.669 INFO:tasks.rados.rados.0.mira097.stdout:1533: done (0 left) 2014-08-22 19:30:54,669.669 INFO:tasks.rados.rados.0.mira097.stdout:1534: write oid 134 current snap is 110 2014-08-22 19:30:54,669.669 INFO:tasks.rados.rados.0.mira097.stdout:1534: seq_num 855 ranges {706805=676872,2152575=154230} 2014-08-22 19:30:54,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1534: writing mira09725123-134 from 706805 to 1383677 tid 1 2014-08-22 19:30:54,707.707 INFO:tasks.rados.rados.0.mira097.stdout:1534: writing mira09725123-134 from 2152575 to 2306805 tid 2 2014-08-22 19:30:54,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1535: read oid 211 snap -1 2014-08-22 19:30:54,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1536: write oid 87 current snap is 110 2014-08-22 19:30:54,708.708 INFO:tasks.rados.rados.0.mira097.stdout:1536: seq_num 856 ranges {651635=401125,1583061=445269,2621418=430217} 2014-08-22 19:30:54,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1536: writing mira09725123-87 from 651635 to 1052760 tid 1 2014-08-22 19:30:54,745.745 INFO:tasks.rados.rados.0.mira097.stdout:1536: writing mira09725123-87 from 1583061 to 2028330 tid 2 2014-08-22 19:30:54,779.779 INFO:tasks.rados.rados.0.mira097.stdout:1536: writing mira09725123-87 from 2621418 to 3051635 tid 3 2014-08-22 19:30:54,779.779 INFO:tasks.rados.rados.0.mira097.stdout:1537: read oid 274 snap -1 2014-08-22 19:30:54,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1538: snap_remove snap 108 2014-08-22 19:30:54,840.840 INFO:tasks.rados.rados.0.mira097.stdout:1535: expect (ObjNum 503 snap 3 seq_num 503) 2014-08-22 19:30:58,490.490 INFO:tasks.rados.rados.0.mira097.stdout:1535: done (4 left) 2014-08-22 19:30:58,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1538: done (3 left) 2014-08-22 19:30:58,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1539: snap_remove snap 107 2014-08-22 19:30:58,678.678 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4] out_osds: [3, 5, 1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:30:58,679.679 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:30:58,679.679 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.242767 2014-08-22 19:30:58,679.679 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.242767316343' 2014-08-22 19:30:58,681.681 INFO:tasks.rados.rados.0.mira097.stdout:1534: finishing write tid 1 to mira09725123-134 2014-08-22 19:30:58,682.682 INFO:tasks.rados.rados.0.mira097.stdout:1534: finishing write tid 2 to mira09725123-134 2014-08-22 19:30:58,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1534: finishing write tid 4 to mira09725123-134 2014-08-22 19:30:58,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1534: finishing write tid 5 to mira09725123-134 2014-08-22 19:30:58,696.696 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 134 v 2084 (ObjNum 855 snap 110 seq_num 855) dirty exists 2014-08-22 19:30:58,703.703 INFO:tasks.rados.rados.0.mira097.stdout:1537: expect (ObjNum 487 snap 0 seq_num 487) 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.028904 7f0c0b5a6700 1 librados: starting msgr at :/0 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.028908 7f0c0b5a6700 1 librados: starting objecter 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.029059 7f0c0b5a6700 1 librados: setting wanted keys 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.029065 7f0c0b5a6700 1 librados: calling monclient init 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.030006 7f0bfbfff700 10 client.?.objecter ms_handle_connect 0x7f0c040355e0 2014-08-22 19:30:59,129.129 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.030035 7f0bfbfff700 10 client.?.objecter resend_mon_ops 2014-08-22 19:30:59,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.031570 7f0c0b5a6700 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:30:59,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.031656 7f0c0b5a6700 1 librados: init done 2014-08-22 19:30:59,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032010 7f0bfbfff700 10 client.4272.objecter ms_dispatch 0x7f0c04000f80 osd_map(279..279 src has 1..279) v3 2014-08-22 19:30:59,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032020 7f0bfbfff700 3 client.4272.objecter handle_osd_map got epochs [279,279] > 0 2014-08-22 19:30:59,131.131 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032022 7f0bfbfff700 3 client.4272.objecter handle_osd_map decoding full epoch 279 2014-08-22 19:30:59,231.231 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032097 7f0bfbfff700 20 client.4272.objecter dump_active .. 0 homeless 2014-08-22 19:30:59,232.232 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032127 7f0bfbfff700 10 client.4272.objecter ms_dispatch 0x7f0c04000f80 osd_map(279..279 src has 1..279) v3 2014-08-22 19:30:59,232.232 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032130 7f0bfbfff700 3 client.4272.objecter handle_osd_map ignoring epochs [279,279] <= 279 2014-08-22 19:30:59,232.232 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.032132 7f0bfbfff700 20 client.4272.objecter dump_active .. 0 homeless 2014-08-22 19:30:59,527.527 INFO:teuthology.orchestra.run.mira097.stderr:set osd.0 primary-affinity to 0.242767 (8159092) 2014-08-22 19:30:59,531.531 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.431630 7f0c0b5a6700 20 client.4272.objecter shutdown clearing up homeless session... 2014-08-22 19:30:59,532.532 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:52.432112 7f0c0b5a6700 1 librados: shutdown 2014-08-22 19:30:59,718.718 INFO:tasks.rados.rados.0.mira097.stdout:1536: finishing write tid 1 to mira09725123-87 2014-08-22 19:30:59,718.718 INFO:tasks.rados.rados.0.mira097.stdout:1536: finishing write tid 2 to mira09725123-87 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1536: finishing write tid 3 to mira09725123-87 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1536: finishing write tid 5 to mira09725123-87 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1536: finishing write tid 6 to mira09725123-87 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 87 v 2057 (ObjNum 856 snap 110 seq_num 856) dirty exists 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1534: done (3 left) 2014-08-22 19:30:59,719.719 INFO:tasks.rados.rados.0.mira097.stdout:1536: done (2 left) 2014-08-22 19:30:59,720.720 INFO:tasks.rados.rados.0.mira097.stdout:1537: done (1 left) 2014-08-22 19:30:59,720.720 INFO:tasks.rados.rados.0.mira097.stdout:1539: done (0 left) 2014-08-22 19:30:59,720.720 INFO:tasks.rados.rados.0.mira097.stdout:1540: delete oid 153 current snap is 110 2014-08-22 19:30:59,736.736 INFO:tasks.rados.rados.0.mira097.stdout:1540: done (0 left) 2014-08-22 19:30:59,737.737 INFO:tasks.rados.rados.0.mira097.stdout:1541: write oid 29 current snap is 110 2014-08-22 19:30:59,737.737 INFO:tasks.rados.rados.0.mira097.stdout:1541: seq_num 858 ranges {7021=1} 2014-08-22 19:30:59,737.737 INFO:tasks.rados.rados.0.mira097.stdout:1541: writing mira09725123-29 from 7021 to 7022 tid 1 2014-08-22 19:30:59,737.737 INFO:tasks.rados.rados.0.mira097.stdout:1542: write oid 338 current snap is 110 2014-08-22 19:30:59,737.737 INFO:tasks.rados.rados.0.mira097.stdout:1542: seq_num 859 ranges {751851=1} 2014-08-22 19:30:59,748.748 INFO:tasks.rados.rados.0.mira097.stdout:1542: writing mira09725123-338 from 751851 to 751852 tid 1 2014-08-22 19:30:59,749.749 INFO:tasks.rados.rados.0.mira097.stdout:1543: read oid 448 snap -1 2014-08-22 19:30:59,749.749 INFO:tasks.rados.rados.0.mira097.stdout:1544: snap_remove snap 102 2014-08-22 19:30:59,782.782 INFO:tasks.rados.rados.0.mira097.stdout:1543: expect (ObjNum 759 snap 69 seq_num 759) 2014-08-22 19:31:01,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1542: finishing write tid 1 to mira09725123-338 2014-08-22 19:31:01,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1542: finishing write tid 3 to mira09725123-338 2014-08-22 19:31:01,790.790 INFO:tasks.rados.rados.0.mira097.stdout:1542: finishing write tid 4 to mira09725123-338 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 338 v 2073 (ObjNum 859 snap 110 seq_num 859) dirty exists 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1542: done (3 left) 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1543: done (2 left) 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1544: done (1 left) 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1545: read oid 1 snap -1 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1541: finishing write tid 1 to mira09725123-29 2014-08-22 19:31:01,791.791 INFO:tasks.rados.rados.0.mira097.stdout:1541: finishing write tid 3 to mira09725123-29 2014-08-22 19:31:01,792.792 INFO:tasks.rados.rados.0.mira097.stdout:1541: finishing write tid 4 to mira09725123-29 2014-08-22 19:31:01,792.792 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 29 v 2066 (ObjNum 858 snap 110 seq_num 858) dirty exists 2014-08-22 19:31:01,792.792 INFO:tasks.rados.rados.0.mira097.stdout:1541: done (1 left) 2014-08-22 19:31:01,792.792 INFO:tasks.rados.rados.0.mira097.stdout:1546: delete oid 473 current snap is 110 2014-08-22 19:31:01,841.841 INFO:tasks.rados.rados.0.mira097.stdout:1545: expect (ObjNum 0 snap 0 seq_num 0) 2014-08-22 19:31:02,008.008 INFO:tasks.rados.rados.0.mira097.stdout:1545: done (1 left) 2014-08-22 19:31:02,008.008 INFO:tasks.rados.rados.0.mira097.stdout:1546: done (0 left) 2014-08-22 19:31:02,009.009 INFO:tasks.rados.rados.0.mira097.stdout:1547: read oid 297 snap -1 2014-08-22 19:31:02,009.009 INFO:tasks.rados.rados.0.mira097.stdout:1548: snap_remove snap 100 2014-08-22 19:31:02,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1547: done (1 left) 2014-08-22 19:31:02,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1548: done (0 left) 2014-08-22 19:31:02,520.520 INFO:tasks.rados.rados.0.mira097.stdout:1549: copy_from oid 429 from oid 463 current snap is 110 2014-08-22 19:31:02,521.521 INFO:tasks.rados.rados.0.mira097.stdout:1550: write oid 374 current snap is 110 2014-08-22 19:31:02,521.521 INFO:tasks.rados.rados.0.mira097.stdout:1550: seq_num 861 ranges {143334=1} 2014-08-22 19:31:02,521.521 INFO:tasks.rados.rados.0.mira097.stdout:1550: writing mira09725123-374 from 143334 to 143335 tid 1 2014-08-22 19:31:02,521.521 INFO:tasks.rados.rados.0.mira097.stdout:1551: write oid 254 current snap is 110 2014-08-22 19:31:02,521.521 INFO:tasks.rados.rados.0.mira097.stdout:1551: seq_num 862 ranges {453093=419471,1609165=769289,2453092=1} 2014-08-22 19:31:02,544.544 INFO:tasks.rados.rados.0.mira097.stdout:1551: writing mira09725123-254 from 453093 to 872564 tid 1 2014-08-22 19:31:02,576.576 INFO:tasks.rados.rados.0.mira097.stdout:1551: writing mira09725123-254 from 1609165 to 2378454 tid 2 2014-08-22 19:31:02,577.577 INFO:tasks.rados.rados.0.mira097.stdout:1551: writing mira09725123-254 from 2453092 to 2453093 tid 3 2014-08-22 19:31:02,577.577 INFO:tasks.rados.rados.0.mira097.stdout:1552: write oid 368 current snap is 110 2014-08-22 19:31:02,578.578 INFO:tasks.rados.rados.0.mira097.stdout:1552: seq_num 863 ranges {548323=675947,1854497=293826} 2014-08-22 19:31:02,611.611 INFO:tasks.rados.rados.0.mira097.stdout:1552: writing mira09725123-368 from 548323 to 1224270 tid 1 2014-08-22 19:31:02,624.624 INFO:tasks.rados.rados.0.mira097.stdout:1552: writing mira09725123-368 from 1854497 to 2148323 tid 2 2014-08-22 19:31:02,626.626 INFO:tasks.rados.rados.0.mira097.stdout:1553: write oid 329 current snap is 110 2014-08-22 19:31:02,627.627 INFO:tasks.rados.rados.0.mira097.stdout:1553: seq_num 864 ranges {644577=799176,1844576=1} 2014-08-22 19:31:02,656.656 INFO:tasks.rados.rados.0.mira097.stdout:1553: writing mira09725123-329 from 644577 to 1443753 tid 1 2014-08-22 19:31:02,694.694 INFO:tasks.rados.rados.0.mira097.stdout:1553: writing mira09725123-329 from 1844576 to 1844577 tid 2 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1554: read oid 497 snap -1 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1550: finishing write tid 1 to mira09725123-374 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1550: finishing write tid 3 to mira09725123-374 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1555: delete oid 324 current snap is 110 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:1550: finishing write tid 4 to mira09725123-374 2014-08-22 19:31:02,695.695 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 374 v 2105 (ObjNum 861 snap 110 seq_num 861) dirty exists 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1551: finishing write tid 1 to mira09725123-254 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1551: finishing write tid 2 to mira09725123-254 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1551: finishing write tid 3 to mira09725123-254 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1551: finishing write tid 5 to mira09725123-254 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1552: finishing write tid 1 to mira09725123-368 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1552: finishing write tid 2 to mira09725123-368 2014-08-22 19:31:02,696.696 INFO:tasks.rados.rados.0.mira097.stdout:1552: finishing write tid 4 to mira09725123-368 2014-08-22 19:31:02,697.697 INFO:tasks.rados.rados.0.mira097.stdout:1550: done (6 left) 2014-08-22 19:31:02,697.697 INFO:tasks.rados.rados.0.mira097.stdout:1555: done (5 left) 2014-08-22 19:31:02,697.697 INFO:tasks.rados.rados.0.mira097.stdout:1556: write oid 231 current snap is 110 2014-08-22 19:31:02,697.697 INFO:tasks.rados.rados.0.mira097.stdout:1556: seq_num 866 ranges {648540=494208,1592477=685141,2923078=510092,3848539=1} 2014-08-22 19:31:02,716.716 INFO:tasks.rados.rados.0.mira097.stdout:1556: writing mira09725123-231 from 648540 to 1142748 tid 1 2014-08-22 19:31:02,748.748 INFO:tasks.rados.rados.0.mira097.stdout:1556: writing mira09725123-231 from 1592477 to 2277618 tid 2 2014-08-22 19:31:02,760.760 INFO:tasks.rados.rados.0.mira097.stdout:1556: writing mira09725123-231 from 2923078 to 3433170 tid 3 2014-08-22 19:31:02,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1556: writing mira09725123-231 from 3848539 to 3848540 tid 4 2014-08-22 19:31:02,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1557: copy_from oid 369 from oid 212 current snap is 110 2014-08-22 19:31:02,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1558: write oid 61 current snap is 110 2014-08-22 19:31:02,763.763 INFO:tasks.rados.rados.0.mira097.stdout:1558: seq_num 867 ranges {744794=750685,1991282=506760,3162448=382346} 2014-08-22 19:31:02,795.795 INFO:tasks.rados.rados.0.mira097.stdout:1558: writing mira09725123-61 from 744794 to 1495479 tid 1 2014-08-22 19:31:02,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1558: writing mira09725123-61 from 1991282 to 2498042 tid 2 2014-08-22 19:31:02,832.832 INFO:tasks.rados.rados.0.mira097.stdout:1558: writing mira09725123-61 from 3162448 to 3544794 tid 3 2014-08-22 19:31:02,834.834 INFO:tasks.rados.rados.0.mira097.stdout:1559: copy_from oid 171 from oid 48 current snap is 110 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1551: finishing write tid 6 to mira09725123-254 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 254 v 2101 (ObjNum 862 snap 110 seq_num 862) dirty exists 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 1 to mira09725123-231 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 2 to mira09725123-231 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1552: finishing write tid 5 to mira09725123-368 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 368 v 2115 (ObjNum 863 snap 110 seq_num 863) dirty exists 2014-08-22 19:31:02,838.838 INFO:tasks.rados.rados.0.mira097.stdout:1554: expect (ObjNum 633 snap 37 seq_num 633) 2014-08-22 19:31:03,814.814 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 3 to mira09725123-231 2014-08-22 19:31:03,814.814 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 4 to mira09725123-231 2014-08-22 19:31:03,814.814 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 6 to mira09725123-231 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1556: finishing write tid 7 to mira09725123-231 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 231 v 2112 (ObjNum 866 snap 110 seq_num 866) dirty exists 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1551: done (8 left) 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1552: done (7 left) 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1554: done (6 left) 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1556: done (5 left) 2014-08-22 19:31:03,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1560: delete oid 393 current snap is 110 2014-08-22 19:31:03,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1559: finishing copy_from to mira09725123-171 2014-08-22 19:31:03,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1559: got expected ENOENT (src dne) 2014-08-22 19:31:03,816.816 INFO:tasks.rados.rados.0.mira097.stdout:1559: finishing copy_from racing read to mira09725123-171 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1553: finishing write tid 1 to mira09725123-329 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1553: finishing write tid 2 to mira09725123-329 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1553: finishing write tid 4 to mira09725123-329 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1558: finishing write tid 1 to mira09725123-61 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1558: finishing write tid 2 to mira09725123-61 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1558: finishing write tid 3 to mira09725123-61 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1558: finishing write tid 5 to mira09725123-61 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:1557: finishing copy_from to mira09725123-369 2014-08-22 19:31:03,817.817 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 369 v 209 (ObjNum 211 snap 0 seq_num 211) dirty exists 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1553: finishing write tid 5 to mira09725123-329 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 329 v 2095 (ObjNum 864 snap 110 seq_num 864) dirty exists 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1549: finishing copy_from to mira09725123-429 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 429 v 2040 (ObjNum 462 snap 0 seq_num 462) dirty exists 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1558: finishing write tid 6 to mira09725123-61 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 61 v 2107 (ObjNum 867 snap 110 seq_num 867) dirty exists 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1557: finishing copy_from racing read to mira09725123-369 2014-08-22 19:31:03,818.818 INFO:tasks.rados.rados.0.mira097.stdout:1549: finishing copy_from racing read to mira09725123-429 2014-08-22 19:31:03,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1549: done (5 left) 2014-08-22 19:31:03,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1553: done (4 left) 2014-08-22 19:31:03,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1557: done (3 left) 2014-08-22 19:31:03,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1558: done (2 left) 2014-08-22 19:31:03,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1559: done (1 left) 2014-08-22 19:31:03,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1560: done (0 left) 2014-08-22 19:31:03,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1561: delete oid 393 current snap is 110 2014-08-22 19:31:03,871.871 INFO:tasks.rados.rados.0.mira097.stdout:1561: done (0 left) 2014-08-22 19:31:03,871.871 INFO:tasks.rados.rados.0.mira097.stdout:1562: read oid 335 snap -1 2014-08-22 19:31:03,871.871 INFO:tasks.rados.rados.0.mira097.stdout:1563: read oid 447 snap -1 2014-08-22 19:31:03,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1564: write oid 122 current snap is 110 2014-08-22 19:31:03,873.873 INFO:tasks.rados.rados.0.mira097.stdout:1564: seq_num 870 ranges {728658=1} 2014-08-22 19:31:03,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1564: writing mira09725123-122 from 728658 to 728659 tid 1 2014-08-22 19:31:03,986.986 INFO:tasks.rados.rados.0.mira097.stdout:1565: snap_create 2014-08-22 19:31:03,986.986 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:31:03,987.987 INFO:tasks.rados.rados.0.mira097.stdout:1563: expect (ObjNum 446 snap 0 seq_num 446) 2014-08-22 19:31:04,332.332 INFO:tasks.rados.rados.0.mira097.stdout:1562: expect (ObjNum 757 snap 67 seq_num 757) 2014-08-22 19:31:04,538.538 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4] out_osds: [3, 5, 1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:31:04,539.539 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 2 1' 2014-08-22 19:31:04,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.885895 7fb69544a700 1 librados: starting msgr at :/0 2014-08-22 19:31:04,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.885907 7fb69544a700 1 librados: starting objecter 2014-08-22 19:31:04,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.886104 7fb69544a700 1 librados: setting wanted keys 2014-08-22 19:31:04,987.987 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.886110 7fb69544a700 1 librados: calling monclient init 2014-08-22 19:31:04,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.886896 7fb68e2e8700 10 client.?.objecter ms_handle_connect 0x7fb6900355e0 2014-08-22 19:31:04,988.988 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.886918 7fb68e2e8700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888266 7fb69544a700 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888363 7fb69544a700 1 librados: init done 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888712 7fb68e2e8700 10 client.4259.objecter ms_dispatch 0x7fb690000f80 osd_map(282..282 src has 1..282) v3 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888722 7fb68e2e8700 3 client.4259.objecter handle_osd_map got epochs [282,282] > 0 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888725 7fb68e2e8700 3 client.4259.objecter handle_osd_map decoding full epoch 282 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888832 7fb68e2e8700 20 client.4259.objecter dump_active .. 0 homeless 2014-08-22 19:31:04,989.989 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888880 7fb68e2e8700 10 client.4259.objecter ms_dispatch 0x7fb690000f80 osd_map(282..282 src has 1..282) v3 2014-08-22 19:31:04,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888886 7fb68e2e8700 3 client.4259.objecter handle_osd_map ignoring epochs [282,282] <= 282 2014-08-22 19:31:04,990.990 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:57.888889 7fb68e2e8700 20 client.4259.objecter dump_active .. 0 homeless 2014-08-22 19:31:05,471.471 INFO:tasks.rados.rados.0.mira097.stdout:1564: finishing write tid 1 to mira09725123-122 2014-08-22 19:31:05,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1564: finishing write tid 3 to mira09725123-122 2014-08-22 19:31:05,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1564: finishing write tid 4 to mira09725123-122 2014-08-22 19:31:05,472.472 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 122 v 2322 (ObjNum 870 snap 110 seq_num 870) dirty exists 2014-08-22 19:31:05,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1562: done (2 left) 2014-08-22 19:31:05,472.472 INFO:tasks.rados.rados.0.mira097.stdout:1563: done (1 left) 2014-08-22 19:31:05,473.473 INFO:tasks.rados.rados.0.mira097.stdout:1564: done (0 left) 2014-08-22 19:31:06,077.077 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.2 to 1 (8655362) 2014-08-22 19:31:06,077.077 INFO:tasks.rados.rados.0.mira097.stdout:1565: done (0 left) 2014-08-22 19:31:06,077.077 INFO:tasks.rados.rados.0.mira097.stdout:1566: write oid 457 current snap is 111 2014-08-22 19:31:06,078.078 INFO:tasks.rados.rados.0.mira097.stdout:1566: seq_num 871 ranges {638417=673998,2072770=538965,3038416=1} 2014-08-22 19:31:06,086.086 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:58.986906 7fb69544a700 20 client.4259.objecter shutdown clearing up homeless session... 2014-08-22 19:31:06,089.089 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:58.989847 7fb69544a700 1 librados: shutdown 2014-08-22 19:31:06,097.097 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 0 1' 2014-08-22 19:31:06,107.107 INFO:tasks.rados.rados.0.mira097.stdout:1566: writing mira09725123-457 from 638417 to 1312415 tid 1 2014-08-22 19:31:06,129.129 INFO:tasks.rados.rados.0.mira097.stdout:1566: writing mira09725123-457 from 2072770 to 2611735 tid 2 2014-08-22 19:31:06,253.253 INFO:tasks.rados.rados.0.mira097.stdout:1566: writing mira09725123-457 from 3038416 to 3038417 tid 3 2014-08-22 19:31:06,253.253 INFO:tasks.rados.rados.0.mira097.stdout:1567: write oid 359 current snap is 111 2014-08-22 19:31:06,253.253 INFO:tasks.rados.rados.0.mira097.stdout:1567: seq_num 872 ranges {734670=797226,2186901=547769} 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1567: writing mira09725123-359 from 734670 to 1531896 tid 1 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1567: writing mira09725123-359 from 2186901 to 2734670 tid 2 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1568: rollback oid 249 current snap is 111 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 249 to 84 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1566: finishing write tid 1 to mira09725123-457 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1566: finishing write tid 2 to mira09725123-457 2014-08-22 19:31:06,254.254 INFO:tasks.rados.rados.0.mira097.stdout:1566: finishing write tid 3 to mira09725123-457 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1566: finishing write tid 5 to mira09725123-457 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1566: finishing write tid 6 to mira09725123-457 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 457 v 2333 (ObjNum 871 snap 111 seq_num 871) dirty exists 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1566: done (2 left) 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1569: write oid 361 current snap is 111 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1569: seq_num 873 ranges {429900=769031,1900011=429756,2429899=1} 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1569: writing mira09725123-361 from 429900 to 1198931 tid 1 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1569: writing mira09725123-361 from 1900011 to 2329767 tid 2 2014-08-22 19:31:06,255.255 INFO:tasks.rados.rados.0.mira097.stdout:1569: writing mira09725123-361 from 2429899 to 2429900 tid 3 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1570: read oid 296 snap -1 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 249 v 2124 (ObjNum 248 snap 0 seq_num 248) dirty exists 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1568: done (3 left) 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1571: write oid 446 current snap is 111 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1571: seq_num 874 ranges {773706=493283,1730495=783599,3173705=1} 2014-08-22 19:31:06,256.256 INFO:tasks.rados.rados.0.mira097.stdout:1571: writing mira09725123-446 from 773706 to 1266989 tid 1 2014-08-22 19:31:06,283.283 INFO:tasks.rados.rados.0.mira097.stdout:1571: writing mira09725123-446 from 1730495 to 2514094 tid 2 2014-08-22 19:31:06,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1571: writing mira09725123-446 from 3173705 to 3173706 tid 3 2014-08-22 19:31:06,288.288 INFO:tasks.rados.rados.0.mira097.stdout:1572: snap_remove snap 91 2014-08-22 19:31:06,289.289 INFO:tasks.rados.rados.0.mira097.stdout:1569: finishing write tid 1 to mira09725123-361 2014-08-22 19:31:06,289.289 INFO:tasks.rados.rados.0.mira097.stdout:1567: finishing write tid 1 to mira09725123-359 2014-08-22 19:31:06,289.289 INFO:tasks.rados.rados.0.mira097.stdout:1567: finishing write tid 2 to mira09725123-359 2014-08-22 19:31:06,290.290 INFO:tasks.rados.rados.0.mira097.stdout:1567: finishing write tid 4 to mira09725123-359 2014-08-22 19:31:06,290.290 INFO:tasks.rados.rados.0.mira097.stdout:1567: finishing write tid 5 to mira09725123-359 2014-08-22 19:31:06,290.290 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 359 v 2124 (ObjNum 872 snap 111 seq_num 872) dirty exists 2014-08-22 19:31:06,290.290 INFO:tasks.rados.rados.0.mira097.stdout:1569: finishing write tid 2 to mira09725123-361 2014-08-22 19:31:06,290.290 INFO:tasks.rados.rados.0.mira097.stdout:1569: finishing write tid 3 to mira09725123-361 2014-08-22 19:31:06,300.300 INFO:tasks.rados.rados.0.mira097.stdout:1569: finishing write tid 5 to mira09725123-361 2014-08-22 19:31:06,300.300 INFO:tasks.rados.rados.0.mira097.stdout:1569: finishing write tid 6 to mira09725123-361 2014-08-22 19:31:06,300.300 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 361 v 2133 (ObjNum 873 snap 111 seq_num 873) dirty exists 2014-08-22 19:31:06,300.300 INFO:tasks.rados.rados.0.mira097.stdout:1571: finishing write tid 1 to mira09725123-446 2014-08-22 19:31:06,301.301 INFO:tasks.rados.rados.0.mira097.stdout:1570: expect (ObjNum 295 snap 0 seq_num 295) 2014-08-22 19:31:06,301.301 INFO:tasks.rados.rados.0.mira097.stdout:1571: finishing write tid 2 to mira09725123-446 2014-08-22 19:31:06,301.301 INFO:tasks.rados.rados.0.mira097.stdout:1571: finishing write tid 3 to mira09725123-446 2014-08-22 19:31:06,302.302 INFO:tasks.rados.rados.0.mira097.stdout:1571: finishing write tid 5 to mira09725123-446 2014-08-22 19:31:06,304.304 INFO:tasks.rados.rados.0.mira097.stdout:1571: finishing write tid 6 to mira09725123-446 2014-08-22 19:31:06,304.304 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 446 v 2131 (ObjNum 874 snap 111 seq_num 874) dirty exists 2014-08-22 19:31:06,573.573 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.472116 7f122c85a700 1 librados: starting msgr at :/0 2014-08-22 19:31:06,573.573 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.472123 7f122c85a700 1 librados: starting objecter 2014-08-22 19:31:06,574.574 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.472459 7f122c85a700 1 librados: setting wanted keys 2014-08-22 19:31:06,574.574 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.472473 7f122c85a700 1 librados: calling monclient init 2014-08-22 19:31:06,574.574 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.474033 7f12258ad700 10 client.?.objecter ms_handle_connect 0x7f12280355e0 2014-08-22 19:31:06,575.575 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.474074 7f12258ad700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:06,577.577 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.476968 7f122c85a700 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:06,577.577 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477168 7f122c85a700 1 librados: init done 2014-08-22 19:31:06,578.578 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477690 7f12258ad700 10 client.4275.objecter ms_dispatch 0x7f1228000f80 osd_map(283..283 src has 1..283) v3 2014-08-22 19:31:06,578.578 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477708 7f12258ad700 3 client.4275.objecter handle_osd_map got epochs [283,283] > 0 2014-08-22 19:31:06,579.579 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477715 7f12258ad700 3 client.4275.objecter handle_osd_map decoding full epoch 283 2014-08-22 19:31:06,579.579 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477885 7f12258ad700 20 client.4275.objecter dump_active .. 0 homeless 2014-08-22 19:31:06,579.579 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477948 7f12258ad700 10 client.4275.objecter ms_dispatch 0x7f1228000f80 osd_map(283..283 src has 1..283) v3 2014-08-22 19:31:06,579.579 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477955 7f12258ad700 3 client.4275.objecter handle_osd_map ignoring epochs [283,283] <= 283 2014-08-22 19:31:06,579.579 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:30:59.477958 7f12258ad700 20 client.4275.objecter dump_active .. 0 homeless 2014-08-22 19:31:07,089.089 INFO:tasks.rados.rados.0.mira097.stdout:1567: done (4 left) 2014-08-22 19:31:07,090.090 INFO:tasks.rados.rados.0.mira097.stdout:1569: done (3 left) 2014-08-22 19:31:07,090.090 INFO:tasks.rados.rados.0.mira097.stdout:1570: done (2 left) 2014-08-22 19:31:07,090.090 INFO:tasks.rados.rados.0.mira097.stdout:1571: done (1 left) 2014-08-22 19:31:07,090.090 INFO:tasks.rados.rados.0.mira097.stdout:1572: done (0 left) 2014-08-22 19:31:07,091.091 INFO:tasks.rados.rados.0.mira097.stdout:1573: snap_remove snap 84 2014-08-22 19:31:07,129.129 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.0 to 1 (8655362) 2014-08-22 19:31:07,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.030338 7f122c85a700 20 client.4275.objecter shutdown clearing up homeless session... 2014-08-22 19:31:07,130.130 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.031036 7f122c85a700 1 librados: shutdown 2014-08-22 19:31:07,138.138 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd reweight 4 1' 2014-08-22 19:31:07,783.783 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.682756 7f9f93c41700 1 librados: starting msgr at :/0 2014-08-22 19:31:07,783.783 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.682763 7f9f93c41700 1 librados: starting objecter 2014-08-22 19:31:07,784.784 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.683046 7f9f93c41700 1 librados: setting wanted keys 2014-08-22 19:31:07,784.784 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.683059 7f9f93c41700 1 librados: calling monclient init 2014-08-22 19:31:07,785.785 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.685191 7f9f90cee700 10 client.?.objecter ms_handle_connect 0x7f9f8c0355e0 2014-08-22 19:31:07,785.785 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.685288 7f9f90cee700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:07,790.790 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.689601 7f9f93c41700 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:07,790.790 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.689797 7f9f93c41700 1 librados: init done 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690750 7f9f90cee700 10 client.4276.objecter ms_dispatch 0x7f9f8c000f80 osd_map(284..284 src has 1..284) v3 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690764 7f9f90cee700 3 client.4276.objecter handle_osd_map got epochs [284,284] > 0 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690767 7f9f90cee700 3 client.4276.objecter handle_osd_map decoding full epoch 284 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690907 7f9f90cee700 20 client.4276.objecter dump_active .. 0 homeless 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690968 7f9f90cee700 10 client.4276.objecter ms_dispatch 0x7f9f8c000f80 osd_map(284..284 src has 1..284) v3 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690973 7f9f90cee700 3 client.4276.objecter handle_osd_map ignoring epochs [284,284] <= 284 2014-08-22 19:31:07,791.791 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:00.690975 7f9f90cee700 20 client.4276.objecter dump_active .. 0 homeless 2014-08-22 19:31:08,099.099 INFO:tasks.rados.rados.0.mira097.stdout:1573: done (0 left) 2014-08-22 19:31:08,100.100 INFO:tasks.rados.rados.0.mira097.stdout:1574: read oid 152 snap 97 2014-08-22 19:31:08,100.100 INFO:tasks.rados.rados.0.mira097.stdout:1575: copy_from oid 205 from oid 136 current snap is 111 2014-08-22 19:31:08,100.100 INFO:tasks.rados.rados.0.mira097.stdout:1576: read oid 51 snap -1 2014-08-22 19:31:08,101.101 INFO:tasks.rados.rados.0.mira097.stdout:1577: delete oid 418 current snap is 111 2014-08-22 19:31:08,141.141 INFO:teuthology.orchestra.run.mira097.stderr:reweighted osd.4 to 1 (8655362) 2014-08-22 19:31:08,147.147 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.044874 7f9f93c41700 20 client.4276.objecter shutdown clearing up homeless session... 2014-08-22 19:31:08,149.149 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.049789 7f9f93c41700 1 librados: shutdown 2014-08-22 19:31:08,157.157 INFO:tasks.ceph.ceph_manager:waiting for recovery to complete 2014-08-22 19:31:08,157.157 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:31:08,172.172 INFO:tasks.rados.rados.0.mira097.stdout:1574: done (3 left) 2014-08-22 19:31:08,172.172 INFO:tasks.rados.rados.0.mira097.stdout:1576: done (2 left) 2014-08-22 19:31:08,173.173 INFO:tasks.rados.rados.0.mira097.stdout:1577: done (1 left) 2014-08-22 19:31:08,173.173 INFO:tasks.rados.rados.0.mira097.stdout:1578: read oid 173 snap -1 2014-08-22 19:31:08,177.177 INFO:tasks.rados.rados.0.mira097.stdout:1579: write oid 58 current snap is 111 2014-08-22 19:31:08,177.177 INFO:tasks.rados.rados.0.mira097.stdout:1579: seq_num 876 ranges {130116=1} 2014-08-22 19:31:08,177.177 INFO:tasks.rados.rados.0.mira097.stdout:1579: writing mira09725123-58 from 130116 to 130117 tid 1 2014-08-22 19:31:08,178.178 INFO:tasks.rados.rados.0.mira097.stdout:1580: write oid 438 current snap is 111 2014-08-22 19:31:08,178.178 INFO:tasks.rados.rados.0.mira097.stdout:1580: seq_num 877 ranges {474947=400000} 2014-08-22 19:31:08,273.273 INFO:tasks.rados.rados.0.mira097.stdout:1580: writing mira09725123-438 from 474947 to 874947 tid 1 2014-08-22 19:31:08,273.273 INFO:tasks.rados.rados.0.mira097.stdout:1581: delete oid 84 current snap is 111 2014-08-22 19:31:08,273.273 INFO:tasks.rados.rados.0.mira097.stdout:1579: finishing write tid 1 to mira09725123-58 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1579: finishing write tid 3 to mira09725123-58 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1580: finishing write tid 1 to mira09725123-438 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1580: finishing write tid 3 to mira09725123-438 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1579: finishing write tid 4 to mira09725123-58 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 58 v 2171 (ObjNum 876 snap 111 seq_num 876) dirty exists 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:1580: finishing write tid 4 to mira09725123-438 2014-08-22 19:31:08,274.274 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 438 v 2176 (ObjNum 877 snap 111 seq_num 877) dirty exists 2014-08-22 19:31:08,285.285 INFO:tasks.rados.rados.0.mira097.stdout:1578: expect (ObjNum 172 snap 0 seq_num 172) 2014-08-22 19:31:08,605.605 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.504867 7fea95e3c700 1 librados: starting msgr at :/0 2014-08-22 19:31:08,605.605 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.504873 7fea95e3c700 1 librados: starting objecter 2014-08-22 19:31:08,606.606 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.505020 7fea95e3c700 1 librados: setting wanted keys 2014-08-22 19:31:08,606.606 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.505024 7fea95e3c700 1 librados: calling monclient init 2014-08-22 19:31:08,606.606 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.505772 7fea8e7fc700 10 client.?.objecter ms_handle_connect 0x7fea900355e0 2014-08-22 19:31:08,606.606 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.505794 7fea8e7fc700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:08,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507298 7fea95e3c700 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:08,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507412 7fea95e3c700 1 librados: init done 2014-08-22 19:31:08,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507638 7fea8e7fc700 10 client.4262.objecter ms_dispatch 0x7fea90000f80 osd_map(285..285 src has 1..285) v3 2014-08-22 19:31:08,607.607 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507647 7fea8e7fc700 3 client.4262.objecter handle_osd_map got epochs [285,285] > 0 2014-08-22 19:31:08,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507650 7fea8e7fc700 3 client.4262.objecter handle_osd_map decoding full epoch 285 2014-08-22 19:31:08,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507740 7fea8e7fc700 20 client.4262.objecter dump_active .. 0 homeless 2014-08-22 19:31:08,608.608 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507769 7fea8e7fc700 10 client.4262.objecter ms_dispatch 0x7fea90000f80 osd_map(285..285 src has 1..285) v3 2014-08-22 19:31:08,609.609 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507771 7fea8e7fc700 3 client.4262.objecter handle_osd_map ignoring epochs [285,285] <= 285 2014-08-22 19:31:08,609.609 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.507772 7fea8e7fc700 20 client.4262.objecter dump_active .. 0 homeless 2014-08-22 19:31:08,786.786 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:31:09,141.141 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.952104 7fea95e3c700 20 client.4262.objecter shutdown clearing up homeless session... 2014-08-22 19:31:09,141.141 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:01.952841 7fea95e3c700 1 librados: shutdown 2014-08-22 19:31:09,327.327 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-08-22 19:31:09,767.767 INFO:tasks.rados.rados.0.mira097.stdout:1575: finishing copy_from to mira09725123-205 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 205 v 1281 (ObjNum 135 snap 0 seq_num 135) dirty exists 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1575: finishing copy_from racing read to mira09725123-205 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1575: done (4 left) 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1578: done (3 left) 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1579: done (2 left) 2014-08-22 19:31:09,768.768 INFO:tasks.rados.rados.0.mira097.stdout:1580: done (1 left) 2014-08-22 19:31:09,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1581: done (0 left) 2014-08-22 19:31:09,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1582: rollback oid 298 current snap is 111 2014-08-22 19:31:09,769.769 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 298 to 104 2014-08-22 19:31:09,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1583: write oid 346 current snap is 111 2014-08-22 19:31:09,769.769 INFO:tasks.rados.rados.0.mira097.stdout:1583: seq_num 879 ranges {515007=424497,1315006=1} 2014-08-22 19:31:09,794.794 INFO:tasks.rados.rados.0.mira097.stdout:1583: writing mira09725123-346 from 515007 to 939504 tid 1 2014-08-22 19:31:09,798.798 INFO:tasks.rados.rados.0.mira097.stdout:1583: writing mira09725123-346 from 1315006 to 1315007 tid 2 2014-08-22 19:31:09,798.798 INFO:tasks.rados.rados.0.mira097.stdout:1584: copy_from oid 183 from oid 283 current snap is 111 2014-08-22 19:31:09,798.798 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 298 v 2190 (ObjNum 297 snap 0 seq_num 297) dirty exists 2014-08-22 19:31:09,798.798 INFO:tasks.rados.rados.0.mira097.stdout:1582: done (2 left) 2014-08-22 19:31:09,799.799 INFO:tasks.rados.rados.0.mira097.stdout:1585: write oid 159 current snap is 111 2014-08-22 19:31:09,799.799 INFO:tasks.rados.rados.0.mira097.stdout:1585: seq_num 880 ranges {575164=547725,1640247=707618,2575163=1} 2014-08-22 19:31:09,823.823 INFO:tasks.rados.rados.0.mira097.stdout:1585: writing mira09725123-159 from 575164 to 1122889 tid 1 2014-08-22 19:31:09,851.851 INFO:tasks.rados.rados.0.mira097.stdout:1585: writing mira09725123-159 from 1640247 to 2347865 tid 2 2014-08-22 19:31:09,855.855 INFO:tasks.rados.rados.0.mira097.stdout:1585: writing mira09725123-159 from 2575163 to 2575164 tid 3 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1586: snap_create 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1583: finishing write tid 1 to mira09725123-346 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1583: finishing write tid 2 to mira09725123-346 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1583: finishing write tid 4 to mira09725123-346 2014-08-22 19:31:09,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1583: finishing write tid 5 to mira09725123-346 2014-08-22 19:31:09,859.859 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 346 v 2194 (ObjNum 879 snap 111 seq_num 879) dirty exists 2014-08-22 19:31:09,859.859 INFO:tasks.rados.rados.0.mira097.stdout:1584: finishing copy_from to mira09725123-183 2014-08-22 19:31:09,859.859 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 183 v 1848 (ObjNum 813 snap 83 seq_num 813) dirty exists 2014-08-22 19:31:09,859.859 INFO:tasks.rados.rados.0.mira097.stdout:1584: finishing copy_from racing read to mira09725123-183 2014-08-22 19:31:09,869.869 INFO:tasks.rados.rados.0.mira097.stdout:1583: done (2 left) 2014-08-22 19:31:09,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1584: done (1 left) 2014-08-22 19:31:09,870.870 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:31:09,870.870 INFO:tasks.rados.rados.0.mira097.stdout:1585: finishing write tid 1 to mira09725123-159 2014-08-22 19:31:09,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1585: finishing write tid 2 to mira09725123-159 2014-08-22 19:31:09,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1585: finishing write tid 3 to mira09725123-159 2014-08-22 19:31:09,884.884 INFO:tasks.rados.rados.0.mira097.stdout:1585: finishing write tid 5 to mira09725123-159 2014-08-22 19:31:09,890.890 INFO:tasks.rados.rados.0.mira097.stdout:1585: finishing write tid 6 to mira09725123-159 2014-08-22 19:31:09,891.891 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 159 v 2362 (ObjNum 880 snap 111 seq_num 880) dirty exists 2014-08-22 19:31:09,891.891 INFO:tasks.rados.rados.0.mira097.stdout:1585: done (0 left) 2014-08-22 19:31:09,966.966 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.864780 7f121a453700 1 librados: starting msgr at :/0 2014-08-22 19:31:09,966.966 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.864788 7f121a453700 1 librados: starting objecter 2014-08-22 19:31:09,966.966 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.865068 7f121a453700 1 librados: setting wanted keys 2014-08-22 19:31:09,966.966 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.865082 7f121a453700 1 librados: calling monclient init 2014-08-22 19:31:09,967.967 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.866809 7f1212ffd700 10 client.?.objecter ms_handle_connect 0x7f12140355e0 2014-08-22 19:31:09,967.967 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.866854 7f1212ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:09,970.970 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.869911 7f121a453700 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:09,970.970 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870108 7f121a453700 1 librados: init done 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870538 7f1212ffd700 10 client.4278.objecter ms_dispatch 0x7f1214000f80 osd_map(285..285 src has 1..285) v3 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870550 7f1212ffd700 3 client.4278.objecter handle_osd_map got epochs [285,285] > 0 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870553 7f1212ffd700 3 client.4278.objecter handle_osd_map decoding full epoch 285 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870695 7f1212ffd700 20 client.4278.objecter dump_active .. 0 homeless 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870748 7f1212ffd700 10 client.4278.objecter ms_dispatch 0x7f1214000f80 osd_map(285..285 src has 1..285) v3 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870753 7f1212ffd700 3 client.4278.objecter handle_osd_map ignoring epochs [285,285] <= 285 2014-08-22 19:31:09,971.971 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:02.870755 7f1212ffd700 20 client.4278.objecter dump_active .. 0 homeless 2014-08-22 19:31:10,088.088 INFO:teuthology.orchestra.run.mira097.stderr:dumped all in format json 2014-08-22 19:31:10,252.252 INFO:tasks.rados.rados.0.mira097.stdout:1586: done (0 left) 2014-08-22 19:31:10,252.252 INFO:tasks.rados.rados.0.mira097.stdout:1587: snap_remove snap 106 2014-08-22 19:31:10,316.316 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:03.187526 7f121a453700 20 client.4278.objecter shutdown clearing up homeless session... 2014-08-22 19:31:10,316.316 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:03.188478 7f121a453700 1 librados: shutdown 2014-08-22 19:31:10,654.654 INFO:teuthology.orchestra.run.mira097:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-08-22 19:31:11,327.327 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.226389 7fb89a002700 1 librados: starting msgr at :/0 2014-08-22 19:31:11,327.327 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.226395 7fb89a002700 1 librados: starting objecter 2014-08-22 19:31:11,327.327 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.226601 7fb89a002700 1 librados: setting wanted keys 2014-08-22 19:31:11,328.328 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.226607 7fb89a002700 1 librados: calling monclient init 2014-08-22 19:31:11,328.328 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.228405 7fb892ffd700 10 client.?.objecter ms_handle_connect 0x7fb8940355e0 2014-08-22 19:31:11,328.328 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.228478 7fb892ffd700 10 client.?.objecter resend_mon_ops 2014-08-22 19:31:11,331.331 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.231529 7fb89a002700 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2014-08-22 19:31:11,331.331 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.231625 7fb89a002700 1 librados: init done 2014-08-22 19:31:11,333.333 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232574 7fb892ffd700 10 client.4279.objecter ms_dispatch 0x7fb894000f80 osd_map(286..286 src has 1..286) v3 2014-08-22 19:31:11,333.333 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232582 7fb892ffd700 3 client.4279.objecter handle_osd_map got epochs [286,286] > 0 2014-08-22 19:31:11,333.333 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232584 7fb892ffd700 3 client.4279.objecter handle_osd_map decoding full epoch 286 2014-08-22 19:31:11,333.333 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232657 7fb892ffd700 20 client.4279.objecter dump_active .. 0 homeless 2014-08-22 19:31:11,334.334 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232835 7fb892ffd700 10 client.4279.objecter ms_dispatch 0x7fb894000f80 osd_map(286..286 src has 1..286) v3 2014-08-22 19:31:11,334.334 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232845 7fb892ffd700 3 client.4279.objecter handle_osd_map ignoring epochs [286,286] <= 286 2014-08-22 19:31:11,334.334 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.232847 7fb892ffd700 20 client.4279.objecter dump_active .. 0 homeless 2014-08-22 19:31:11,413.413 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.313106 7fb89a002700 20 client.4279.objecter shutdown clearing up homeless session... 2014-08-22 19:31:11,413.413 INFO:teuthology.orchestra.run.mira097.stderr:2014-08-22 11:31:04.313723 7fb89a002700 1 librados: shutdown 2014-08-22 19:31:11,420.420 INFO:tasks.ceph.ceph_manager:{u'election_epoch': 4, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 0, u'epoch': 1, u'by_rank': [], u'up': 0, u'in': 0}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.136.128:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.137.130:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.136.128:6790/0'}], u'modified': u'2014-08-22 11:20:54.537903', u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81', u'created': u'2014-08-22 11:20:54.537903'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 4, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003310', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.001678', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-08-22 11:30:36.481117', u'name': u'a', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873710332, u'health': u'HEALTH_OK', u'kb_used': 23070720, u'store_stats': {u'bytes_total': 11162869, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 68598, u'bytes_sst': 10111231}}, {u'last_updated': u'2014-08-22 11:30:36.471334', u'name': u'b', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 876837892, u'health': u'HEALTH_OK', u'kb_used': 19943160, u'store_stats': {u'bytes_total': 12211454, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 68598, u'bytes_sst': 10111240}}, {u'last_updated': u'2014-08-22 11:30:36.074082', u'name': u'c', u'avail_percent': 92, u'kb_total': 944797076, u'kb_avail': 873711192, u'health': u'HEALTH_OK', u'kb_used': 23069860, u'store_stats': {u'bytes_total': 12211454, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 68598, u'bytes_sst': 10111240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 94 > pgp_num 84'}, {u'severity': u'HEALTH_WARN', u'summary': u'mon.b has mon_osd_down_out_interval set to 0'}]}, u'pgmap': {u'bytes_total': 2999149522944, u'num_pgs': 102, u'data_bytes': 1232747597, u'bytes_used': 3033653248, u'op_per_sec': 48, u'version': 469, u'pgs_by_state': [{u'count': 102, u'state_name': u'active+clean'}], u'read_bytes_sec': 3573803, u'write_bytes_sec': 7871593, u'bytes_avail': 2996115869696}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': False, u'nearfull': False, u'num_osds': 6, u'num_up_osds': 6, u'epoch': 286, u'num_in_osds': 3}}, u'fsid': u'e92480aa-fdf7-4143-a1da-fa6625fbfe81'} 2014-08-22 19:31:11,421.421 INFO:tasks.ceph.ceph_manager:recovered! 2014-08-22 19:31:11,421.421 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:31:11,421.421 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-22 19:31:11,421.421 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:31:11,421.421 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:31:11,422.422 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-22 19:31:11,663.663 INFO:tasks.rados.rados.0.mira097.stdout:1587: done (0 left) 2014-08-22 19:31:11,663.663 INFO:tasks.rados.rados.0.mira097.stdout:1588: delete oid 376 current snap is 112 2014-08-22 19:31:11,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1588: done (0 left) 2014-08-22 19:31:11,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1589: write oid 226 current snap is 112 2014-08-22 19:31:11,734.734 INFO:tasks.rados.rados.0.mira097.stdout:1589: seq_num 882 ranges {615224=492358,1682016=599432,2967389=47835} 2014-08-22 19:31:11,756.756 INFO:tasks.rados.rados.0.mira097.stdout:1589: writing mira09725123-226 from 615224 to 1107582 tid 1 2014-08-22 19:31:11,779.779 INFO:tasks.rados.rados.0.mira097.stdout:1589: writing mira09725123-226 from 1682016 to 2281448 tid 2 2014-08-22 19:31:11,786.786 INFO:tasks.rados.rados.0.mira097.stdout:1589: writing mira09725123-226 from 2967389 to 3015224 tid 3 2014-08-22 19:31:11,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1590: write oid 188 current snap is 112 2014-08-22 19:31:11,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1590: seq_num 883 ranges {559030=747810,1927348=553275,3184486=574544} 2014-08-22 19:31:11,815.815 INFO:tasks.rados.rados.0.mira097.stdout:1590: writing mira09725123-188 from 559030 to 1306840 tid 1 2014-08-22 19:31:11,835.835 INFO:tasks.rados.rados.0.mira097.stdout:1590: writing mira09725123-188 from 1927348 to 2480623 tid 2 2014-08-22 19:31:11,858.858 INFO:tasks.rados.rados.0.mira097.stdout:1590: writing mira09725123-188 from 3184486 to 3759030 tid 3 2014-08-22 19:31:11,861.861 INFO:tasks.rados.rados.0.mira097.stdout:1591: delete oid 477 current snap is 112 2014-08-22 19:31:11,861.861 INFO:tasks.rados.rados.0.mira097.stdout:1589: finishing write tid 1 to mira09725123-226 2014-08-22 19:31:11,861.861 INFO:tasks.rados.rados.0.mira097.stdout:1589: finishing write tid 2 to mira09725123-226 2014-08-22 19:31:11,861.861 INFO:tasks.rados.rados.0.mira097.stdout:1589: finishing write tid 3 to mira09725123-226 2014-08-22 19:31:11,862.862 INFO:tasks.rados.rados.0.mira097.stdout:1589: finishing write tid 5 to mira09725123-226 2014-08-22 19:31:11,862.862 INFO:tasks.rados.rados.0.mira097.stdout:1589: finishing write tid 6 to mira09725123-226 2014-08-22 19:31:11,862.862 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 226 v 2192 (ObjNum 882 snap 112 seq_num 882) dirty exists 2014-08-22 19:31:11,876.876 INFO:tasks.rados.rados.0.mira097.stdout:1590: finishing write tid 1 to mira09725123-188 2014-08-22 19:31:11,885.885 INFO:tasks.rados.rados.0.mira097.stdout:1590: finishing write tid 2 to mira09725123-188 2014-08-22 19:31:11,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1589: done (2 left) 2014-08-22 19:31:11,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1591: done (1 left) 2014-08-22 19:31:11,892.892 INFO:tasks.rados.rados.0.mira097.stdout:1592: read oid 244 snap -1 2014-08-22 19:31:11,893.893 INFO:tasks.rados.rados.0.mira097.stdout:1593: read oid 126 snap -1 2014-08-22 19:31:11,893.893 INFO:tasks.rados.rados.0.mira097.stdout:1594: copy_from oid 433 from oid 114 current snap is 112 2014-08-22 19:31:11,893.893 INFO:tasks.rados.rados.0.mira097.stdout:1590: finishing write tid 3 to mira09725123-188 2014-08-22 19:31:11,893.893 INFO:tasks.rados.rados.0.mira097.stdout:1595: copy_from oid 220 from oid 496 current snap is 112 2014-08-22 19:31:11,894.894 INFO:tasks.rados.rados.0.mira097.stdout:1596: snap_create 2014-08-22 19:31:11,894.894 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 5 2014-08-22 19:31:11,894.894 INFO:tasks.rados.rados.0.mira097.stdout:1592: done (4 left) 2014-08-22 19:31:11,894.894 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:31:11,896.896 INFO:tasks.rados.rados.0.mira097.stdout:1590: finishing write tid 5 to mira09725123-188 2014-08-22 19:31:11,898.898 INFO:tasks.rados.rados.0.mira097.stdout:1590: finishing write tid 6 to mira09725123-188 2014-08-22 19:31:11,898.898 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 188 v 2167 (ObjNum 883 snap 112 seq_num 883) dirty exists 2014-08-22 19:31:11,898.898 INFO:tasks.rados.rados.0.mira097.stdout:1590: done (3 left) 2014-08-22 19:31:11,898.898 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 3 2014-08-22 19:31:11,922.922 INFO:tasks.rados.rados.0.mira097.stdout:1593: expect (ObjNum 161 snap 0 seq_num 161) 2014-08-22 19:31:12,535.535 INFO:tasks.rados.rados.0.mira097.stdout:1595: finishing copy_from to mira09725123-220 2014-08-22 19:31:12,535.535 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 220 v 680 (ObjNum 495 snap 0 seq_num 495) dirty exists 2014-08-22 19:31:12,535.535 INFO:tasks.rados.rados.0.mira097.stdout:1595: finishing copy_from racing read to mira09725123-220 2014-08-22 19:31:12,535.535 INFO:tasks.rados.rados.0.mira097.stdout:1594: finishing copy_from to mira09725123-433 2014-08-22 19:31:12,535.535 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 433 v 129 (ObjNum 113 snap 0 seq_num 113) dirty exists 2014-08-22 19:31:12,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1594: finishing copy_from racing read to mira09725123-433 2014-08-22 19:31:12,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1593: done (2 left) 2014-08-22 19:31:12,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1594: done (1 left) 2014-08-22 19:31:12,536.536 INFO:tasks.rados.rados.0.mira097.stdout:1595: done (0 left) 2014-08-22 19:31:12,681.681 INFO:tasks.rados.rados.0.mira097.stdout:1596: done (0 left) 2014-08-22 19:31:12,681.681 INFO:tasks.rados.rados.0.mira097.stdout:1597: rollback oid 51 current snap is 113 2014-08-22 19:31:12,681.681 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 51 to 97 2014-08-22 19:31:12,682.682 INFO:tasks.rados.rados.0.mira097.stdout:1598: copy_from oid 50 from oid 19 current snap is 113 2014-08-22 19:31:12,682.682 INFO:tasks.rados.rados.0.mira097.stdout:1599: write oid 25 current snap is 113 2014-08-22 19:31:12,682.682 INFO:tasks.rados.rados.0.mira097.stdout:1599: seq_num 885 ranges {564017=443867,1685468=78549} 2014-08-22 19:31:12,703.703 INFO:tasks.rados.rados.0.mira097.stdout:1599: writing mira09725123-25 from 564017 to 1007884 tid 1 2014-08-22 19:31:12,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1599: writing mira09725123-25 from 1685468 to 1764017 tid 2 2014-08-22 19:31:12,713.713 INFO:tasks.rados.rados.0.mira097.stdout:1600: read oid 387 snap -1 2014-08-22 19:31:12,713.713 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 51 v 2028 (ObjNum 258754440 snap 32605 seq_num 32) dirty dne 2014-08-22 19:31:12,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1597: done (3 left) 2014-08-22 19:31:12,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1601: rollback oid 269 current snap is 113 2014-08-22 19:31:12,714.714 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 269 to 96 2014-08-22 19:31:12,714.714 INFO:tasks.rados.rados.0.mira097.stdout:1602: snap_create 2014-08-22 19:31:12,714.714 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 4 2014-08-22 19:31:12,732.732 INFO:tasks.rados.rados.0.mira097.stdout:1600: expect (ObjNum 386 snap 0 seq_num 386) 2014-08-22 19:31:12,777.777 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 269 v 2204 (ObjNum 120 snap 0 seq_num 120) dirty exists 2014-08-22 19:31:12,778.778 INFO:tasks.rados.rados.0.mira097.stdout:1600: done (3 left) 2014-08-22 19:31:12,778.778 INFO:tasks.rados.rados.0.mira097.stdout:1601: done (2 left) 2014-08-22 19:31:12,778.778 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 2 2014-08-22 19:31:14,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1599: finishing write tid 1 to mira09725123-25 2014-08-22 19:31:14,373.373 INFO:tasks.rados.rados.0.mira097.stdout:1599: finishing write tid 2 to mira09725123-25 2014-08-22 19:31:14,374.374 INFO:tasks.rados.rados.0.mira097.stdout:1599: finishing write tid 4 to mira09725123-25 2014-08-22 19:31:14,376.376 INFO:tasks.rados.rados.0.mira097.stdout:1599: finishing write tid 5 to mira09725123-25 2014-08-22 19:31:14,376.376 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 25 v 2187 (ObjNum 885 snap 113 seq_num 885) dirty exists 2014-08-22 19:31:14,376.376 INFO:tasks.rados.rados.0.mira097.stdout:1599: done (1 left) 2014-08-22 19:31:14,376.376 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 1 2014-08-22 19:31:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:1598: finishing copy_from to mira09725123-50 2014-08-22 19:31:14,390.390 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 50 v 903 (ObjNum 585 snap 25 seq_num 585) dirty exists 2014-08-22 19:31:14,391.391 INFO:tasks.rados.rados.0.mira097.stdout:1598: finishing copy_from racing read to mira09725123-50 2014-08-22 19:31:14,391.391 INFO:tasks.rados.rados.0.mira097.stdout:1598: done (0 left) 2014-08-22 19:31:15,401.401 INFO:tasks.rados.rados.0.mira097.stdout:1602: done (0 left) 2014-08-22 19:31:15,402.402 INFO:tasks.rados.rados.0.mira097.stdout:1603: delete oid 199 current snap is 114 2014-08-22 19:31:15,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1603: done (0 left) 2014-08-22 19:31:15,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1604: write oid 272 current snap is 114 2014-08-22 19:31:15,419.419 INFO:tasks.rados.rados.0.mira097.stdout:1604: seq_num 887 ranges {756524=400000} 2014-08-22 19:31:15,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1604: writing mira09725123-272 from 756524 to 1156524 tid 1 2014-08-22 19:31:15,434.434 INFO:tasks.rados.rados.0.mira097.stdout:1605: read oid 199 snap -1 2014-08-22 19:31:15,435.435 INFO:tasks.rados.rados.0.mira097.stdout:1606: write oid 318 current snap is 114 2014-08-22 19:31:15,435.435 INFO:tasks.rados.rados.0.mira097.stdout:1606: seq_num 888 ranges {548906=546800,1875416=407099,2688789=260117} 2014-08-22 19:31:15,457.457 INFO:tasks.rados.rados.0.mira097.stdout:1606: writing mira09725123-318 from 548906 to 1095706 tid 1 2014-08-22 19:31:15,475.475 INFO:tasks.rados.rados.0.mira097.stdout:1606: writing mira09725123-318 from 1875416 to 2282515 tid 2 2014-08-22 19:31:15,487.487 INFO:tasks.rados.rados.0.mira097.stdout:1606: writing mira09725123-318 from 2688789 to 2948906 tid 3 2014-08-22 19:31:15,490.490 INFO:tasks.rados.rados.0.mira097.stdout:1607: read oid 96 snap -1 2014-08-22 19:31:15,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1605: done (3 left) 2014-08-22 19:31:15,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1608: read oid 382 snap 112 2014-08-22 19:31:15,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1609: rollback oid 405 current snap is 114 2014-08-22 19:31:15,491.491 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 405 to 111 2014-08-22 19:31:15,491.491 INFO:tasks.rados.rados.0.mira097.stdout:1610: rollback oid 489 current snap is 114 2014-08-22 19:31:15,492.492 INFO:tasks.rados.rados.0.mira097.stdout:rollback oid 489 to 111 2014-08-22 19:31:15,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1611: read oid 24 snap -1 2014-08-22 19:31:15,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1612: write oid 131 current snap is 114 2014-08-22 19:31:15,492.492 INFO:tasks.rados.rados.0.mira097.stdout:1612: seq_num 889 ranges {761512=518604,1822252=477295,2722719=438793} 2014-08-22 19:31:15,508.508 INFO:tasks.rados.rados.0.mira097.stdout:1612: writing mira09725123-131 from 761512 to 1280116 tid 1 2014-08-22 19:31:15,573.573 INFO:tasks.rados.rados.0.mira097.stdout:1612: writing mira09725123-131 from 1822252 to 2299547 tid 2 2014-08-22 19:31:15,574.574 INFO:tasks.rados.rados.0.mira097.stdout:1612: writing mira09725123-131 from 2722719 to 3161512 tid 3 2014-08-22 19:31:15,574.574 INFO:tasks.rados.rados.0.mira097.stdout:1613: write oid 446 current snap is 114 2014-08-22 19:31:15,574.574 INFO:tasks.rados.rados.0.mira097.stdout:1613: seq_num 890 ranges {455718=641832,1534337=547490,2654121=201597} 2014-08-22 19:31:15,574.574 INFO:tasks.rados.rados.0.mira097.stdout:1613: writing mira09725123-446 from 455718 to 1097550 tid 1 2014-08-22 19:31:15,597.597 INFO:tasks.rados.rados.0.mira097.stdout:1613: writing mira09725123-446 from 1534337 to 2081827 tid 2 2014-08-22 19:31:15,601.601 INFO:tasks.rados.rados.0.mira097.stdout:1613: writing mira09725123-446 from 2654121 to 2855718 tid 3 2014-08-22 19:31:15,601.601 INFO:tasks.rados.rados.0.mira097.stdout:1614: copy_from oid 410 from oid 118 current snap is 114 2014-08-22 19:31:15,608.608 INFO:tasks.rados.rados.0.mira097.stdout:1606: finishing write tid 1 to mira09725123-318 2014-08-22 19:31:15,608.608 INFO:tasks.rados.rados.0.mira097.stdout:1606: finishing write tid 2 to mira09725123-318 2014-08-22 19:31:15,608.608 INFO:tasks.rados.rados.0.mira097.stdout:1606: finishing write tid 3 to mira09725123-318 2014-08-22 19:31:15,608.608 INFO:tasks.rados.rados.0.mira097.stdout:1606: finishing write tid 5 to mira09725123-318 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 405 v 2387 (ObjNum 830 snap 99 seq_num 830) dirty exists 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1604: finishing write tid 1 to mira09725123-272 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1604: finishing write tid 3 to mira09725123-272 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1612: finishing write tid 1 to mira09725123-131 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1612: finishing write tid 2 to mira09725123-131 2014-08-22 19:31:15,609.609 INFO:tasks.rados.rados.0.mira097.stdout:1604: finishing write tid 4 to mira09725123-272 2014-08-22 19:31:15,621.621 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 272 v 2210 (ObjNum 887 snap 114 seq_num 887) dirty exists 2014-08-22 19:31:15,621.621 INFO:tasks.rados.rados.0.mira097.stdout:1612: finishing write tid 3 to mira09725123-131 2014-08-22 19:31:15,621.621 INFO:tasks.rados.rados.0.mira097.stdout:1612: finishing write tid 5 to mira09725123-131 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1613: finishing write tid 1 to mira09725123-446 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1604: done (9 left) 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1609: done (8 left) 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1611: done (7 left) 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1615: read oid 99 snap 110 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1616: copy_from oid 393 from oid 403 current snap is 114 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout:1617: snap_create 2014-08-22 19:31:15,622.622 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 9 2014-08-22 19:31:15,623.623 INFO:tasks.rados.rados.0.mira097.stdout:1607: expect (ObjNum 95 snap 0 seq_num 95) 2014-08-22 19:31:16,785.785 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 489 v 2395 (ObjNum 488 snap 0 seq_num 488) dirty exists 2014-08-22 19:31:16,785.785 INFO:tasks.rados.rados.0.mira097.stdout:1613: finishing write tid 2 to mira09725123-446 2014-08-22 19:31:16,785.785 INFO:tasks.rados.rados.0.mira097.stdout:1613: finishing write tid 3 to mira09725123-446 2014-08-22 19:31:16,786.786 INFO:tasks.rados.rados.0.mira097.stdout:1613: finishing write tid 5 to mira09725123-446 2014-08-22 19:31:16,786.786 INFO:tasks.rados.rados.0.mira097.stdout:1607: done (8 left) 2014-08-22 19:31:16,786.786 INFO:tasks.rados.rados.0.mira097.stdout:1610: done (7 left) 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout: waiting on 7 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1614: finishing copy_from to mira09725123-410 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 410 v 1724 (ObjNum 782 snap 73 seq_num 782) dirty exists 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1612: finishing write tid 6 to mira09725123-131 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 131 v 2219 (ObjNum 889 snap 114 seq_num 889) dirty exists 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1606: finishing write tid 6 to mira09725123-318 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 318 v 2235 (ObjNum 888 snap 114 seq_num 888) dirty exists 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1613: finishing write tid 6 to mira09725123-446 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 446 v 2230 (ObjNum 890 snap 114 seq_num 890) dirty exists 2014-08-22 19:31:16,787.787 INFO:tasks.rados.rados.0.mira097.stdout:1615: expect (ObjNum 98 snap 0 seq_num 98) 2014-08-22 19:31:16,788.788 INFO:tasks.rados.rados.0.mira097.stdout:1608: expect (ObjNum 381 snap 0 seq_num 381) 2014-08-22 19:31:16,921.921 INFO:tasks.thrashosds.thrasher:in_osds: [2, 0, 4] out_osds: [3, 5, 1] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-08-22 19:31:16,921.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-22 19:31:16,922.922 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2014-08-22 19:31:16,922.922 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-08-22 19:31:16,922.922 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-08-22 19:31:16,922.922 INFO:teuthology.orchestra.run.mira114:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-08-22 19:31:17,449.449 INFO:tasks.rados.rados.0.mira097.stdout:1616: finishing copy_from to mira09725123-393 2014-08-22 19:31:17,449.449 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 393 v 1942 (ObjNum 402 snap 0 seq_num 402) dirty exists 2014-08-22 19:31:17,449.449 INFO:tasks.rados.rados.0.mira097.stdout:1614: finishing copy_from racing read to mira09725123-410 2014-08-22 19:31:17,449.449 INFO:tasks.rados.rados.0.mira097.stdout:1616: finishing copy_from racing read to mira09725123-393 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1606: done (6 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1608: done (5 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1612: done (4 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1613: done (3 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1614: done (2 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1615: done (1 left) 2014-08-22 19:31:17,450.450 INFO:tasks.rados.rados.0.mira097.stdout:1616: done (0 left) 2014-08-22 19:31:17,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1617: done (0 left) 2014-08-22 19:31:17,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1618: read oid 271 snap 90 2014-08-22 19:31:17,496.496 INFO:tasks.rados.rados.0.mira097.stdout:1619: snap_remove snap 113 2014-08-22 19:31:17,599.599 INFO:tasks.rados.rados.0.mira097.stdout:1618: expect (ObjNum 270 snap 0 seq_num 270) 2014-08-22 19:31:19,388.388 INFO:tasks.rados.rados.0.mira097.stdout:1618: done (1 left) 2014-08-22 19:31:19,390.390 INFO:tasks.rados.rados.0.mira097.stdout:1619: done (0 left) 2014-08-22 19:31:19,390.390 INFO:tasks.rados.rados.0.mira097.stdout:1620: write oid 241 current snap is 115 2014-08-22 19:31:19,390.390 INFO:tasks.rados.rados.0.mira097.stdout:1620: seq_num 891 ranges {649122=1} 2014-08-22 19:31:19,398.398 INFO:tasks.rados.rados.0.mira097.stdout:1620: writing mira09725123-241 from 649122 to 649123 tid 1 2014-08-22 19:31:19,398.398 INFO:tasks.rados.rados.0.mira097.stdout:1621: read oid 329 snap -1 2014-08-22 19:31:19,399.399 INFO:tasks.rados.rados.0.mira097.stdout:1622: snap_remove snap 104 2014-08-22 19:31:19,403.403 INFO:tasks.rados.rados.0.mira097.stdout:1621: expect (ObjNum 864 snap 110 seq_num 864) 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:1620: finishing write tid 1 to mira09725123-241 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:1620: finishing write tid 3 to mira09725123-241 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:1620: finishing write tid 4 to mira09725123-241 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 241 v 2239 (ObjNum 891 snap 115 seq_num 891) dirty exists 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:1620: done (2 left) 2014-08-22 19:31:20,209.209 INFO:tasks.rados.rados.0.mira097.stdout:1621: done (1 left) 2014-08-22 19:31:20,210.210 INFO:tasks.rados.rados.0.mira097.stdout:1622: done (0 left) 2014-08-22 19:31:20,210.210 INFO:tasks.rados.rados.0.mira097.stdout:1623: write oid 149 current snap is 115 2014-08-22 19:31:20,210.210 INFO:tasks.rados.rados.0.mira097.stdout:1623: seq_num 892 ranges {345376=1} 2014-08-22 19:31:20,214.214 INFO:tasks.rados.rados.0.mira097.stdout:1623: writing mira09725123-149 from 345376 to 345377 tid 1 2014-08-22 19:31:20,214.214 INFO:tasks.rados.rados.0.mira097.stdout:1624: snap_remove snap 96 2014-08-22 19:31:20,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1623: finishing write tid 1 to mira09725123-149 2014-08-22 19:31:20,259.259 INFO:tasks.rados.rados.0.mira097.stdout:1623: finishing write tid 3 to mira09725123-149 2014-08-22 19:31:20,426.426 INFO:tasks.rados.rados.0.mira097.stdout:1623: finishing write tid 4 to mira09725123-149 2014-08-22 19:31:20,426.426 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 149 v 2263 (ObjNum 892 snap 115 seq_num 892) dirty exists 2014-08-22 19:31:20,551.551 INFO:tasks.rados.rados.0.mira097.stdout:1623: done (1 left) 2014-08-22 19:31:20,551.551 INFO:tasks.rados.rados.0.mira097.stdout:1624: done (0 left) 2014-08-22 19:31:20,551.551 INFO:tasks.rados.rados.0.mira097.stdout:1625: write oid 341 current snap is 115 2014-08-22 19:31:20,551.551 INFO:tasks.rados.rados.0.mira097.stdout:1625: seq_num 893 ranges {556958=1} 2014-08-22 19:31:20,558.558 INFO:tasks.rados.rados.0.mira097.stdout:1625: writing mira09725123-341 from 556958 to 556959 tid 1 2014-08-22 19:31:20,559.559 INFO:tasks.rados.rados.0.mira097.stdout:1626: read oid 205 snap 97 2014-08-22 19:31:20,559.559 INFO:tasks.rados.rados.0.mira097.stdout:1627: copy_from oid 406 from oid 326 current snap is 115 2014-08-22 19:31:20,559.559 INFO:tasks.rados.rados.0.mira097.stdout:1628: write oid 499 current snap is 115 2014-08-22 19:31:20,560.560 INFO:tasks.rados.rados.0.mira097.stdout:1628: seq_num 894 ranges {750365=565146,2017874=332491} 2014-08-22 19:31:20,595.595 INFO:tasks.rados.rados.0.mira097.stdout:1628: writing mira09725123-499 from 750365 to 1315511 tid 1 2014-08-22 19:31:20,614.614 INFO:tasks.rados.rados.0.mira097.stdout:1628: writing mira09725123-499 from 2017874 to 2350365 tid 2 2014-08-22 19:31:20,617.617 INFO:tasks.rados.rados.0.mira097.stdout:1629: read oid 401 snap -1 2014-08-22 19:31:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1625: finishing write tid 1 to mira09725123-341 2014-08-22 19:31:20,726.726 INFO:tasks.rados.rados.0.mira097.stdout:1625: finishing write tid 3 to mira09725123-341 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1625: finishing write tid 4 to mira09725123-341 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stdout:update_object_version oid 341 v 2417 (ObjNum 893 snap 115 seq_num 893) dirty exists 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1625: done (4 left) 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1630: delete oid 155 current snap is 115 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stdout:1626: expect (ObjNum 39860448 snap 0 seq_num 3019902368) 2014-08-22 19:31:20,727.727 INFO:tasks.rados.rados.0.mira097.stderr:./test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f5cf3fff700 time 2014-08-22 11:31:13.537484 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr:./test/osd/RadosModel.h: 1063: FAILED assert(!old_value.deleted()) 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: ceph version 0.84-653-g0ec7ded (0ec7ded1e4a5c339a9d2a49999bfaaac9da9f358) 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7f5d1004666f] 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x159d) [0x41c34d] 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 3: (librados::C_AioComplete::finish(int)+0x1d) [0x7f5d0ffa914d] 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 4: (Context::complete(int)+0x9) [0x7f5d0ff854d9] 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 5: (Finisher::finisher_thread_entry()+0x160) [0x7f5d10045840] 2014-08-22 19:31:20,728.728 INFO:tasks.rados.rados.0.mira097.stderr: 6: (()+0x7e9a) [0x7f5d0fbe8e9a] 2014-08-22 19:31:20,729.729 INFO:tasks.rados.rados.0.mira097.stderr: 7: (clone()+0x6d) [0x7f5d0f3ff3fd] 2014-08-22 19:31:20,729.729 INFO:tasks.rados.rados.0.mira097.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.