Ceph

2013-08-06T19:19:00.289 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:00.295 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 588 16 7403 7387 50.2441 32 0.707152 1.27163
2013-08-06T19:19:00.318 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:00.318 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph -s'
2013-08-06T19:19:00.623 INFO:teuthology.task.thrashosds.ceph_manager: cluster 3e717ca8-bd17-439a-a1cd-80770f68c0a6
health HEALTH_OK
monmap e1: 3 mons at {a=10.214.133.31:6789/0,b=10.214.133.35:6789/0,c=10.214.133.31:6790/0}, election epoch 8, quorum 0,1,2 a,b,c
osdmap e38: 6 osds: 6 up, 4 in
pgmap v275: 102 pgs: 102 active+clean; 45452 MB data, 62311 MB used, 2727 GB / 2794 GB avail; 78386KB/s wr, 19op/s
mdsmap e5: 1/1/1 up {0=a=up:active}

2013-08-06T19:19:00.623 INFO:teuthology.task.thrashosds.ceph_manager:clean!
2013-08-06T19:19:00.623 INFO:teuthology.task.thrashosds.thrasher:Recovered, killing an osd
2013-08-06T19:19:00.623 INFO:teuthology.task.thrashosds.thrasher:Killing osd 1, live_osds are [1, 3, 2, 5, 4, 0]
2013-08-06T19:19:00.624 DEBUG:teuthology.task.ceph.osd.1:waiting for process to exit
2013-08-06T19:19:00.694 INFO:teuthology.task.ceph.osd.1:Stopped
2013-08-06T19:19:00.695 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:00.695 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph osd down 1'
2013-08-06T19:19:00.695 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.35]: 2013-08-06 19:19:49.697563 7f51d41df700 0 -- 10.214.133.35:0/1018686 >> 10.214.133.31:6800/15075 pipe(0x7f51d0016280 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f51d001de00).fault
2013-08-06T19:19:01.294 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 589 16 7420 7404 50.2743 68 0.16186 1.27114
2013-08-06T19:19:01.416 INFO:teuthology.orchestra.run.err:[10.214.133.31]: marked down osd.1.
2013-08-06T19:19:01.428 INFO:teuthology.task.thrashosds.thrasher:Removing osd 1, in_osds are: [2, 3, 1, 4]
2013-08-06T19:19:01.428 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:01.428 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph osd out 1'
2013-08-06T19:19:02.294 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 590 16 7428 7412 50.2433 32 0.788347 1.27108
2013-08-06T19:19:02.429 INFO:teuthology.orchestra.run.err:[10.214.133.31]: marked out osd.1.
2013-08-06T19:19:02.439 INFO:teuthology.task.thrashosds.thrasher:Waiting for clean again
2013-08-06T19:19:02.439 INFO:teuthology.task.thrashosds.ceph_manager:waiting for clean
2013-08-06T19:19:02.440 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:02.440 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:02.741 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:02.754 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:02.754 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:03.023 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:03.045 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:03.045 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph -s'
2013-08-06T19:19:03.294 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 591 16 7437 7421 50.2192 36 0.158046 1.27007
2013-08-06T19:19:03.298 INFO:teuthology.task.thrashosds.ceph_manager: cluster 3e717ca8-bd17-439a-a1cd-80770f68c0a6
health HEALTH_WARN 25 pgs stale
monmap e1: 3 mons at {a=10.214.133.31:6789/0,b=10.214.133.35:6789/0,c=10.214.133.31:6790/0}, election epoch 8, quorum 0,1,2 a,b,c
osdmap e40: 6 osds: 5 up, 3 in
pgmap v277: 102 pgs: 77 active+clean, 25 stale+active+clean; 45452 MB data, 48367 MB used, 2276 GB / 2328 GB avail
mdsmap e5: 1/1/1 up {0=a=up:active}

2013-08-06T19:19:03.298 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:03.298 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:03.764 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:04.294 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 592 16 7440 7424 50.1546 12 3.88858 1.27099
2013-08-06T19:19:05.294 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 593 16 7457 7441 50.1847 68 0.667485 1.27231
2013-08-06T19:19:06.295 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 594 16 7465 7449 50.1541 32 0.547518 1.27189
2013-08-06T19:19:06.772 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:06.773 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:07.001 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:07.021 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:07.021 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph -s'
2013-08-06T19:19:07.274 INFO:teuthology.task.thrashosds.ceph_manager: cluster 3e717ca8-bd17-439a-a1cd-80770f68c0a6
health HEALTH_WARN 25 pgs stale
monmap e1: 3 mons at {a=10.214.133.31:6789/0,b=10.214.133.35:6789/0,c=10.214.133.31:6790/0}, election epoch 8, quorum 0,1,2 a,b,c
osdmap e41: 6 osds: 5 up, 3 in
pgmap v278: 102 pgs: 77 active+clean, 25 stale+active+clean; 45452 MB data, 48367 MB used, 2276 GB / 2328 GB avail
mdsmap e5: 1/1/1 up {0=a=up:active}

2013-08-06T19:19:07.275 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:07.275 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:07.295 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 595 16 7481 7465 50.1773 64 1.35989 1.2742
2013-08-06T19:19:07.681 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:08.296 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 596 16 7486 7470 50.1267 20 1.72277 1.27431
2013-08-06T19:19:09.295 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 597 16 7502 7486 50.1499 64 0.636611 1.27468
2013-08-06T19:19:10.295 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.35]: 598 16 7506 7490 50.0928 16 1.57742 1.27478
2013-08-06T19:19:10.689 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:10.689 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph pg dump --format=json'
2013-08-06T19:19:10.906 INFO:teuthology.orchestra.run.err:[10.214.133.31]: dumped all in format json
2013-08-06T19:19:10.923 DEBUG:teuthology.misc:with jobid basedir: 98004
2013-08-06T19:19:10.923 DEBUG:teuthology.orchestra.run:Running [10.214.133.31]: '/home/ubuntu/cephtest/98004/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/98004/archive/coverage ceph -s'
2013-08-06T19:19:11.180 INFO:teuthology.task.thrashosds.ceph_manager: cluster 3e717ca8-bd17-439a-a1cd-80770f68c0a6
health HEALTH_WARN 5 pgs backfilling; 5 pgs degraded; 1 pgs incomplete; 12 pgs recovering; 1 pgs stuck inactive; 1 pgs stuck unclean; recovery 7986/22978 degraded (34.755%); recovering 2 o/s, 13500KB/s

ubuntu@teuthology:/a/teuthology-2013-08-06_01:00:25-rados-next-testing-basic-plana/98004