2014-04-01 15:37:06,192.192 DEBUG:teuthology.report:No results_server set in /home/loic/.teuthology.yaml; not attempting to push results 2014-04-01 15:37:06,200.200 DEBUG:teuthology.run:Config: machine_type: plana os_type: ubuntu overrides: admin_socket: branch: master ceph: conf: global: ms inject socket failures: 5000 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 fs: ext4 log-whitelist: - slow request sha1: e672c52b4f8b945a516f2eec006e33665a08f045 ceph-deploy: branch: dev: master conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: e672c52b4f8b945a516f2eec006e33665a08f045 s3tests: branch: master workunit: sha1: e672c52b4f8b945a516f2eec006e33665a08f045 owner: scheduled_teuthology@teuthology roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - client.0 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - client.1 targets: ubuntu@plana78.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZxXbzgCaQZsjUFXMXBP6gCnCguoRfZ8dxbV71vaRpMX8p7avh1cFB2qeXDQMBDYukxJZdVBSEomkW/bPygRa+/vlvj7XhcOroy7SkpVY9ZAFkUKScANybxglH2qo+7TFGNB0Yq9GPBQ6z/3sbB50Uzod6gKPrAhGS/uggrSj2KbXK5UUrUQNsSvMtAJqYPDDWRWtwMH/Ezr76FEM//cveTXoViNh557CgJVB9HqmKHTtk9c9iPIi4DBdH5cGWe2f1Lh3HHVO4KOEepI9/2ItMIhoODEobvNJi62Vh76ZR2lcQJw4njyv3GF4c4khK47xQx6vPttvPxMLprClEM/J1 ubuntu@plana79.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGJX9/cZFlm+ll32X716yKrmR/RE94iH9TusIFLY5vLoE8CBCupjAKdkn4mqOB5eNvwqapMG63Vww5Cl9zo0wKPEHi3jsZCwbAxByc9sBFivZeBnmTUrYNesQvAs1Izr49/4h71oCt98hfX3hl81iEhIEGjjj7XoD3blyYHlyR9LBNoYMllLfi9Pw4KD1snEikGKlR8zMyAgndlZ5ODU4usiCMXLypAa4wFKfR6w17nZI2/Q2xkUy7l59oap50bOR9mSNMOPqpN1KgsK717JCKMU4jHHw+zN0oWvQPmDZK2ckzbyftBfpOqKyidGHpLjpXfuQ1Xk4R9fgUkpE6gcD5 tasks: - chef: null - clock.check: null - install: null - ceph: log-whitelist: - wrongly marked me down - objects unfound and apparently lost - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 - radosbench: clients: - client.0 ec_pool: true time: 1800 unique_pool: true teuthology_branch: master verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.16939 2014-04-01 15:37:06,200.200 INFO:teuthology.run_tasks:Running task internal.save_config... 2014-04-01 15:37:06,200.200 INFO:teuthology.task.internal:Saving configuration 2014-04-01 15:37:06,200.200 INFO:teuthology.run_tasks:Running task internal.check_lock... 2014-04-01 15:37:06,201.201 INFO:teuthology.task.internal:Checking locks... 2014-04-01 15:37:07,048.048 DEBUG:teuthology.task.internal:machine status is {u'sshpubkey': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGJX9/cZFlm+ll32X716yKrmR/RE94iH9TusIFLY5vLoE8CBCupjAKdkn4mqOB5eNvwqapMG63Vww5Cl9zo0wKPEHi3jsZCwbAxByc9sBFivZeBnmTUrYNesQvAs1Izr49/4h71oCt98hfX3hl81iEhIEGjjj7XoD3blyYHlyR9LBNoYMllLfi9Pw4KD1snEikGKlR8zMyAgndlZ5ODU4usiCMXLypAa4wFKfR6w17nZI2/Q2xkUy7l59oap50bOR9mSNMOPqpN1KgsK717JCKMU4jHHw+zN0oWvQPmDZK2ckzbyftBfpOqKyidGHpLjpXfuQ1Xk4R9fgUkpE6gcD5', u'vpshost': None, u'name': u'ubuntu@plana79.front.sepia.ceph.com', u'type': u'plana', u'description': u'None', u'locked_by': u'loic@dachary.org', u'up': 1, u'mac': u'78:2b:cb:55:dc:4e', u'locked': 1, u'distro': u'ubuntu', u'arch': u'x86_64', u'locked_since': u'2014-04-01T05:26:14'} 2014-04-01 15:37:07,454.454 DEBUG:teuthology.task.internal:machine status is {u'sshpubkey': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZxXbzgCaQZsjUFXMXBP6gCnCguoRfZ8dxbV71vaRpMX8p7avh1cFB2qeXDQMBDYukxJZdVBSEomkW/bPygRa+/vlvj7XhcOroy7SkpVY9ZAFkUKScANybxglH2qo+7TFGNB0Yq9GPBQ6z/3sbB50Uzod6gKPrAhGS/uggrSj2KbXK5UUrUQNsSvMtAJqYPDDWRWtwMH/Ezr76FEM//cveTXoViNh557CgJVB9HqmKHTtk9c9iPIi4DBdH5cGWe2f1Lh3HHVO4KOEepI9/2ItMIhoODEobvNJi62Vh76ZR2lcQJw4njyv3GF4c4khK47xQx6vPttvPxMLprClEM/J1', u'vpshost': None, u'name': u'ubuntu@plana78.front.sepia.ceph.com', u'type': u'plana', u'description': u'None', u'locked_by': u'loic@dachary.org', u'up': 1, u'mac': u'78:2b:cb:5b:fc:20', u'locked': 1, u'distro': u'ubuntu', u'arch': u'x86_64', u'locked_since': u'2014-04-01T05:26:14'} 2014-04-01 15:37:07,455.455 INFO:teuthology.run_tasks:Running task internal.connect... 2014-04-01 15:37:07,455.455 INFO:teuthology.task.internal:Opening connections... 2014-04-01 15:37:07,456.456 DEBUG:teuthology.task.internal:connecting to ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:37:09,466.466 DEBUG:teuthology.task.internal:connecting to ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:37:11,378.378 INFO:teuthology.task.internal:roles: ubuntu@plana79.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2014-04-01 15:37:11,379.379 INFO:teuthology.task.internal:roles: ubuntu@plana78.front.sepia.ceph.com - ['mon.b', 'mds.a', 'osd.3', 'osd.4', 'osd.5', 'client.1'] 2014-04-01 15:37:11,379.379 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2014-04-01 15:37:11,379.379 INFO:teuthology.task.internal:Checking for old test directory... 2014-04-01 15:37:11,380.380 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "test '!' -e /home/ubuntu/cephtest" 2014-04-01 15:37:11,701.701 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "test '!' -e /home/ubuntu/cephtest" 2014-04-01 15:37:12,025.025 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2014-04-01 15:37:12,025.025 INFO:teuthology.task.internal:Checking for old /var/lib/ceph... 2014-04-01 15:37:12,026.026 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "test '!' -e /var/lib/ceph" 2014-04-01 15:37:12,347.347 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "test '!' -e /var/lib/ceph" 2014-04-01 15:37:12,711.711 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2014-04-01 15:37:12,712.712 INFO:teuthology.run_tasks:Running task internal.base... 2014-04-01 15:37:12,713.713 INFO:teuthology.task.internal:Creating test directory... 2014-04-01 15:37:12,713.713 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'mkdir -m0755 -- /home/ubuntu/cephtest' 2014-04-01 15:37:13,035.035 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'mkdir -m0755 -- /home/ubuntu/cephtest' 2014-04-01 15:37:13,361.361 INFO:teuthology.run_tasks:Running task internal.archive... 2014-04-01 15:37:13,361.361 INFO:teuthology.task.internal:Creating archive directory... 2014-04-01 15:37:13,362.362 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive' 2014-04-01 15:37:13,683.683 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive' 2014-04-01 15:37:14,017.017 INFO:teuthology.run_tasks:Running task internal.coredump... 2014-04-01 15:37:14,018.018 INFO:teuthology.task.internal:Enabling coredump saving... 2014-04-01 15:37:14,018.018 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core' 2014-04-01 15:37:14,343.343 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core' 2014-04-01 15:37:14,350.350 INFO:teuthology.orchestra.run.out:[10.214.133.38]: kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2014-04-01 15:37:14,675.675 INFO:teuthology.orchestra.run.out:[10.214.133.37]: kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2014-04-01 15:37:14,676.676 INFO:teuthology.run_tasks:Running task internal.sudo... 2014-04-01 15:37:14,676.676 INFO:teuthology.task.internal:Configuring sudo... 2014-04-01 15:37:14,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo sed -i.orig.teuthology -e 's/^\\([^#]*\\) \\(requiretty\\)/\\1 !\\2/g' -e 's/^\\([^#]*\\) !\\(visiblepw\\)/\\1 \\2/g' /etc/sudoers" 2014-04-01 15:37:14,998.998 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo sed -i.orig.teuthology -e 's/^\\([^#]*\\) \\(requiretty\\)/\\1 !\\2/g' -e 's/^\\([^#]*\\) !\\(visiblepw\\)/\\1 \\2/g' /etc/sudoers" 2014-04-01 15:37:15,332.332 INFO:teuthology.run_tasks:Running task internal.syslog... 2014-04-01 15:37:15,332.332 INFO:teuthology.run_tasks:Running task internal.timer... 2014-04-01 15:37:15,333.333 INFO:teuthology.task.internal:Starting timer... 2014-04-01 15:37:15,333.333 INFO:teuthology.run_tasks:Running task chef... 2014-04-01 15:37:15,334.334 INFO:teuthology.task.chef:Running chef-solo... 2014-04-01 15:37:15,334.334 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "wget -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh -x" 2014-04-01 15:37:15,655.655 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "wget -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh -x" 2014-04-01 15:37:15,706.706 INFO:teuthology.orchestra.run.err:[10.214.133.38]: --2014-04-01 06:37:15-- http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD 2014-04-01 15:37:15,707.707 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Resolving ceph.com (ceph.com)... 208.113.241.137, 2607:f298:4:147::b05:fe2a 2014-04-01 15:37:15,708.708 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Connecting to ceph.com (ceph.com)|208.113.241.137|:80... connected. 2014-04-01 15:37:15,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.38]: HTTP request sent, awaiting response... 200 OK 2014-04-01 15:37:15,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Length: unspecified [text/plain] 2014-04-01 15:37:15,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Saving to: `STDOUT' 2014-04-01 15:37:15,800.800 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 15:37:15,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0K ...+ set -e 2014-04-01 15:37:15,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + dirname sh 2014-04-01 15:37:15,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cd . 2014-04-01 15:37:15,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + python -c import platform; distro = platform.dist()[0]; print distro.lower().rstrip() 2014-04-01 15:37:15,802.802 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 163M=0s 2014-04-01 15:37:15,802.802 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 15:37:15,808.808 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:37:15 (163 MB/s) - written to stdout [3313] 2014-04-01 15:37:15,809.809 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 15:37:15,832.832 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + distro=ubuntu 2014-04-01 15:37:15,832.832 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + python -c import platform; distro = platform.dist()[1]; print distro 2014-04-01 15:37:15,866.866 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + distroversion=12.04 2014-04-01 15:37:15,866.866 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + echo ubuntu 2014-04-01 15:37:15,869.869 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + grep -ic -e centos -e redhat -e fedora 2014-04-01 15:37:15,869.869 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + true 2014-04-01 15:37:15,869.869 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + centos=0 2014-04-01 15:37:15,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + + echo ubuntugrep 2014-04-01 15:37:15,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.38]: -ic -e ubuntu -e debian 2014-04-01 15:37:15,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + ubuntu=1 2014-04-01 15:37:15,951.951 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + echo 12.04 2014-04-01 15:37:15,987.987 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + grep -ic 14\.04 2014-04-01 15:37:15,988.988 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + true 2014-04-01 15:37:15,988.988 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + trusty=0 2014-04-01 15:37:15,989.989 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 1 -gt 0 ] 2014-04-01 15:37:15,989.989 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 0 -gt 0 ] 2014-04-01 15:37:15,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + nodepends=--no-install-recommends 2014-04-01 15:37:15,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + dpkg --status -- git 2014-04-01 15:37:15,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + sed -n s/^Status: //p 2014-04-01 15:37:15,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ install ok installed != install ok installed ] 2014-04-01 15:37:15,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ -n ] 2014-04-01 15:37:15,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 0 -gt 0 ] 2014-04-01 15:37:15,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + mktemp -d --tmpdir solo-from-scratch.XXXXXXXXXXXX 2014-04-01 15:37:15,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + SCRATCH=/tmp/solo-from-scratch.PCuqvIiujsq1 2014-04-01 15:37:15,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cd /tmp/solo-from-scratch.PCuqvIiujsq1 2014-04-01 15:37:15,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + trap cleanup INT TERM EXIT 2014-04-01 15:37:15,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + git init 2014-04-01 15:37:15,993.993 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Initialized empty Git repository in /tmp/solo-from-scratch.PCuqvIiujsq1/.git/ 2014-04-01 15:37:15,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + git pull git://ceph.com/ceph-qa-chef.git 2014-04-01 15:37:16,012.012 INFO:teuthology.orchestra.run.err:[10.214.133.37]: --2014-04-01 06:37:15-- http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD 2014-04-01 15:37:16,012.012 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Resolving ceph.com (ceph.com)... 208.113.241.137, 2607:f298:4:147::b05:fe2a 2014-04-01 15:37:16,013.013 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Connecting to ceph.com (ceph.com)|208.113.241.137|:80... connected. 2014-04-01 15:37:16,104.104 INFO:teuthology.orchestra.run.err:[10.214.133.37]: HTTP request sent, awaiting response... 200 OK 2014-04-01 15:37:16,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Length: unspecified [text/plain] 2014-04-01 15:37:16,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Saving to: `STDOUT' 2014-04-01 15:37:16,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 15:37:16,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0K ...+ set -e 2014-04-01 15:37:16,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + dirname sh 2014-04-01 15:37:16,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 215M=0s 2014-04-01 15:37:16,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 15:37:16,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:37:16 (215 MB/s) - written to stdout [3313] 2014-04-01 15:37:16,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 15:37:16,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cd . 2014-04-01 15:37:16,107.107 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + python -c import platform; distro = platform.dist()[0]; print distro.lower().rstrip() 2014-04-01 15:37:16,136.136 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + distro=ubuntu 2014-04-01 15:37:16,137.137 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + python -c import platform; distro = platform.dist()[1]; print distro 2014-04-01 15:37:16,169.169 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + distroversion=12.04 2014-04-01 15:37:16,170.170 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo ubuntu 2014-04-01 15:37:16,170.170 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic -e centos -e redhat -e fedora 2014-04-01 15:37:16,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + true 2014-04-01 15:37:16,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + centos=0 2014-04-01 15:37:16,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo ubuntu 2014-04-01 15:37:16,210.210 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic -e ubuntu -e debian 2014-04-01 15:37:16,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + ubuntu=1 2014-04-01 15:37:16,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo 12.04 2014-04-01 15:37:16,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic 14\.04 2014-04-01 15:37:16,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + true 2014-04-01 15:37:16,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + trusty=0 2014-04-01 15:37:16,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 1 -gt 0 ] 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 0 -gt 0 ] 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + nodepends=--no-install-recommends 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + dpkg --status -- git 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + sed -n s/^Status: //p 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ install ok installed != install ok installed ] 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ -n ] 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 0 -gt 0 ] 2014-04-01 15:37:16,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + mktemp -d --tmpdir solo-from-scratch.XXXXXXXXXXXX 2014-04-01 15:37:16,252.252 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + SCRATCH=/tmp/solo-from-scratch.L3KJTJOfGQkl 2014-04-01 15:37:16,252.252 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cd /tmp/solo-from-scratch.L3KJTJOfGQkl 2014-04-01 15:37:16,252.252 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + trap cleanup INT TERM EXIT 2014-04-01 15:37:16,253.253 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + git init 2014-04-01 15:37:16,282.282 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Initialized empty Git repository in /tmp/solo-from-scratch.L3KJTJOfGQkl/.git/ 2014-04-01 15:37:16,284.284 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + git pull git://ceph.com/ceph-qa-chef.git 2014-04-01 15:37:17,460.460 INFO:teuthology.orchestra.run.err:[10.214.133.38]: From git://ceph.com/ceph-qa-chef 2014-04-01 15:37:17,461.461 INFO:teuthology.orchestra.run.err:[10.214.133.38]: * branch HEAD -> FETCH_HEAD 2014-04-01 15:37:17,523.523 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + ./solo/run 2014-04-01 15:37:17,836.836 INFO:teuthology.orchestra.run.err:[10.214.133.37]: From git://ceph.com/ceph-qa-chef 2014-04-01 15:37:17,837.837 INFO:teuthology.orchestra.run.err:[10.214.133.37]: * branch HEAD -> FETCH_HEAD 2014-04-01 15:37:17,899.899 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + ./solo/run 2014-04-01 15:37:31,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:19-07:00] INFO: Forking chef instance to converge... 2014-04-01 15:37:31,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:19-07:00] INFO: *** Chef 11.8.0 *** 2014-04-01 15:37:31,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:19-07:00] INFO: Chef-client pid: 2643 2014-04-01 15:37:31,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Setting the run_list to ["recipe[ceph-qa::default]"] from JSON 2014-04-01 15:37:31,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Run List is [recipe[ceph-qa::default]] 2014-04-01 15:37:31,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Run List expands to [ceph-qa::default] 2014-04-01 15:37:31,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Starting Chef Run for plana78.front.sepia.ceph.com 2014-04-01 15:37:31,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Running start handlers 2014-04-01 15:37:31,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Start handlers complete. 2014-04-01 15:37:31,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Processing file[/etc/security/limits.d/ubuntu.conf] action create (ceph-qa::default line 8) 2014-04-01 15:37:31,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:20-07:00] INFO: Processing execute[remove ceph packages] action run (ceph-qa::ubuntu line 9) 2014-04-01 15:37:31,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: execute[remove ceph packages] ran successfully 2014-04-01 15:37:31,056.056 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[remove /etc/ceph] action run (ceph-qa::ubuntu line 12) 2014-04-01 15:37:31,056.056 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: execute[remove /etc/ceph] ran successfully 2014-04-01 15:37:31,056.056 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[remove old source entries] action run (ceph-qa::ubuntu line 15) 2014-04-01 15:37:31,056.056 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: execute[remove old source entries] ran successfully 2014-04-01 15:37:31,056.056 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing cookbook_file[/etc/apt/sources.list] action create (ceph-qa::ubuntu line 41) 2014-04-01 15:37:31,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing cookbook_file[/etc/cron.weekly/kernel-clean] action create (ceph-qa::ubuntu line 66) 2014-04-01 15:37:31,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[Restarting Cron] action run (ceph-qa::ubuntu line 72) 2014-04-01 15:37:31,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: execute[Restarting Cron] ran successfully 2014-04-01 15:37:31,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[add autobuild gpg key to apt] action run (ceph-qa::ubuntu line 87) 2014-04-01 15:37:31,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: execute[add autobuild gpg key to apt] ran successfully 2014-04-01 15:37:31,058.058 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: Processing file[/etc/apt/sources.list.d/ceph-extras.list] action create (ceph-qa::ubuntu line 96) 2014-04-01 15:37:31,058.058 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] created file /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 15:37:31,058.058 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] updated file contents /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 15:37:31,058.058 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] owner changed to 0 2014-04-01 15:37:31,058.058 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] group changed to 0 2014-04-01 15:37:31,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] mode changed to 644 2014-04-01 15:37:31,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:22-07:00] INFO: Processing execute[add release gpg key to apt] action run (ceph-qa::ubuntu line 104) 2014-04-01 15:37:31,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:23-07:00] INFO: execute[add release gpg key to apt] ran successfully 2014-04-01 15:37:31,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:23-07:00] INFO: Processing package[python-pip] action install (ceph-qa::ubuntu line 112) 2014-04-01 15:37:31,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing package[python-virtualenv] action install (ceph-qa::ubuntu line 113) 2014-04-01 15:37:31,060.060 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing package[python-dev] action install (ceph-qa::ubuntu line 114) 2014-04-01 15:37:31,060.060 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing package[libevent-dev] action install (ceph-qa::ubuntu line 115) 2014-04-01 15:37:31,060.060 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing package[perl] action install (ceph-qa::ubuntu line 118) 2014-04-01 15:37:31,060.060 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing package[libwww-perl] action install (ceph-qa::ubuntu line 119) 2014-04-01 15:37:31,061.061 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing directory[/home/ubuntu/.cpan/CPAN/] action create (ceph-qa::ubuntu line 121) 2014-04-01 15:37:31,061.061 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing cookbook_file[/home/ubuntu/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 127) 2014-04-01 15:37:31,061.061 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing directory[/root/.cpan/CPAN/] action create (ceph-qa::ubuntu line 134) 2014-04-01 15:37:31,061.061 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing cookbook_file[/root/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 140) 2014-04-01 15:37:31,061.061 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing file[/etc/apt/preferences.d/ceph.pref] action create (ceph-qa::radosgw line 4) 2014-04-01 15:37:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing file[/etc/apt/sources.list.d/radosgw.list] action create (ceph-qa::radosgw line 16) 2014-04-01 15:37:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] created file /etc/apt/sources.list.d/radosgw.list 2014-04-01 15:37:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] updated file contents /etc/apt/sources.list.d/radosgw.list 2014-04-01 15:37:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] owner changed to 0 2014-04-01 15:37:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] group changed to 0 2014-04-01 15:37:31,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] mode changed to 644 2014-04-01 15:37:31,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:26-07:00] INFO: Processing execute[apt-get update] action run (ceph-qa::radosgw line 54) 2014-04-01 15:37:31,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: execute[apt-get update] ran successfully 2014-04-01 15:37:31,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[apache2] action upgrade (ceph-qa::radosgw line 58) 2014-04-01 15:37:31,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[libapache2-mod-fastcgi] action upgrade (ceph-qa::radosgw line 61) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[libfcgi0ldbl] action install (ceph-qa::radosgw line 64) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing service[apache2] action disable (ceph-qa::radosgw line 66) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing service[apache2] action stop (ceph-qa::radosgw line 66) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[lsb-release] action install (ceph-qa::ubuntu line 176) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[build-essential] action install (ceph-qa::ubuntu line 177) 2014-04-01 15:37:31,064.064 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:29-07:00] INFO: Processing package[sysstat] action install (ceph-qa::ubuntu line 178) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[gdb] action install (ceph-qa::ubuntu line 179) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[python-configobj] action install (ceph-qa::ubuntu line 180) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[python-gevent] action install (ceph-qa::ubuntu line 181) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libedit2] action install (ceph-qa::ubuntu line 184) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libssl0.9.8] action install (ceph-qa::ubuntu line 185) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libgoogle-perftools0] action install (ceph-qa::ubuntu line 189) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libboost-thread1.46.1] action install (ceph-qa::ubuntu line 199) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[cryptsetup-bin] action install (ceph-qa::ubuntu line 203) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[xfsprogs] action install (ceph-qa::ubuntu line 204) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[gdisk] action install (ceph-qa::ubuntu line 205) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[parted] action install (ceph-qa::ubuntu line 206) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[smbios-utils] action install (ceph-qa::ubuntu line 210) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libleveldb1] action install (ceph-qa::ubuntu line 211) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libcrypto++9] action install (ceph-qa::ubuntu line 223) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[openmpi-common] action remove (ceph-qa::ubuntu line 234) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libuuid1] action install (ceph-qa::ubuntu line 238) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libfcgi] action install (ceph-qa::ubuntu line 239) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: package[libfcgi] is a virtual package, actually acting on package[libfcgi0ldbl] 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[btrfs-tools] action install (ceph-qa::ubuntu line 240) 2014-04-01 15:37:31,065.065 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libatomic-ops-dev] action install (ceph-qa::ubuntu line 243) 2014-04-01 15:37:31,066.066 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[git-core] action install (ceph-qa::ubuntu line 246) 2014-04-01 15:37:31,066.066 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[attr] action install (ceph-qa::ubuntu line 247) 2014-04-01 15:37:31,216.216 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:19-07:00] INFO: Forking chef instance to converge... 2014-04-01 15:37:31,216.216 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:19-07:00] INFO: *** Chef 11.8.0 *** 2014-04-01 15:37:31,217.217 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:19-07:00] INFO: Chef-client pid: 2630 2014-04-01 15:37:31,217.217 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:20-07:00] INFO: Setting the run_list to ["recipe[ceph-qa::default]"] from JSON 2014-04-01 15:37:31,217.217 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Run List is [recipe[ceph-qa::default]] 2014-04-01 15:37:31,217.217 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Run List expands to [ceph-qa::default] 2014-04-01 15:37:31,218.218 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Starting Chef Run for plana79.front.sepia.ceph.com 2014-04-01 15:37:31,218.218 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Running start handlers 2014-04-01 15:37:31,218.218 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Start handlers complete. 2014-04-01 15:37:31,218.218 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing file[/etc/security/limits.d/ubuntu.conf] action create (ceph-qa::default line 8) 2014-04-01 15:37:31,218.218 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[remove ceph packages] action run (ceph-qa::ubuntu line 9) 2014-04-01 15:37:31,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: execute[remove ceph packages] ran successfully 2014-04-01 15:37:31,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[remove /etc/ceph] action run (ceph-qa::ubuntu line 12) 2014-04-01 15:37:31,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: execute[remove /etc/ceph] ran successfully 2014-04-01 15:37:31,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[remove old source entries] action run (ceph-qa::ubuntu line 15) 2014-04-01 15:37:31,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: execute[remove old source entries] ran successfully 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing cookbook_file[/etc/apt/sources.list] action create (ceph-qa::ubuntu line 41) 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing cookbook_file[/etc/cron.weekly/kernel-clean] action create (ceph-qa::ubuntu line 66) 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[Restarting Cron] action run (ceph-qa::ubuntu line 72) 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: execute[Restarting Cron] ran successfully 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:21-07:00] INFO: Processing execute[add autobuild gpg key to apt] action run (ceph-qa::ubuntu line 87) 2014-04-01 15:37:31,220.220 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: execute[add autobuild gpg key to apt] ran successfully 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: Processing file[/etc/apt/sources.list.d/ceph-extras.list] action create (ceph-qa::ubuntu line 96) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] created file /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] updated file contents /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] owner changed to 0 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] group changed to 0 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] mode changed to 644 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:22-07:00] INFO: Processing execute[add release gpg key to apt] action run (ceph-qa::ubuntu line 104) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:23-07:00] INFO: execute[add release gpg key to apt] ran successfully 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:23-07:00] INFO: Processing package[python-pip] action install (ceph-qa::ubuntu line 112) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing package[python-virtualenv] action install (ceph-qa::ubuntu line 113) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing package[python-dev] action install (ceph-qa::ubuntu line 114) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing package[libevent-dev] action install (ceph-qa::ubuntu line 115) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing package[perl] action install (ceph-qa::ubuntu line 118) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing package[libwww-perl] action install (ceph-qa::ubuntu line 119) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing directory[/home/ubuntu/.cpan/CPAN/] action create (ceph-qa::ubuntu line 121) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing cookbook_file[/home/ubuntu/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 127) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing directory[/root/.cpan/CPAN/] action create (ceph-qa::ubuntu line 134) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing cookbook_file[/root/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 140) 2014-04-01 15:37:31,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing file[/etc/apt/preferences.d/ceph.pref] action create (ceph-qa::radosgw line 4) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing file[/etc/apt/sources.list.d/radosgw.list] action create (ceph-qa::radosgw line 16) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] created file /etc/apt/sources.list.d/radosgw.list 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] updated file contents /etc/apt/sources.list.d/radosgw.list 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] owner changed to 0 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] group changed to 0 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] mode changed to 644 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:26-07:00] INFO: Processing execute[apt-get update] action run (ceph-qa::radosgw line 54) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: execute[apt-get update] ran successfully 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: Processing package[apache2] action upgrade (ceph-qa::radosgw line 58) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: Processing package[libapache2-mod-fastcgi] action upgrade (ceph-qa::radosgw line 61) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: Processing package[libfcgi0ldbl] action install (ceph-qa::radosgw line 64) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: Processing service[apache2] action disable (ceph-qa::radosgw line 66) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:29-07:00] INFO: Processing service[apache2] action stop (ceph-qa::radosgw line 66) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[lsb-release] action install (ceph-qa::ubuntu line 176) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[build-essential] action install (ceph-qa::ubuntu line 177) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[sysstat] action install (ceph-qa::ubuntu line 178) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[gdb] action install (ceph-qa::ubuntu line 179) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[python-configobj] action install (ceph-qa::ubuntu line 180) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[python-gevent] action install (ceph-qa::ubuntu line 181) 2014-04-01 15:37:31,222.222 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libedit2] action install (ceph-qa::ubuntu line 184) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libssl0.9.8] action install (ceph-qa::ubuntu line 185) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libgoogle-perftools0] action install (ceph-qa::ubuntu line 189) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libboost-thread1.46.1] action install (ceph-qa::ubuntu line 199) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[cryptsetup-bin] action install (ceph-qa::ubuntu line 203) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[xfsprogs] action install (ceph-qa::ubuntu line 204) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[gdisk] action install (ceph-qa::ubuntu line 205) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[parted] action install (ceph-qa::ubuntu line 206) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[smbios-utils] action install (ceph-qa::ubuntu line 210) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libleveldb1] action install (ceph-qa::ubuntu line 211) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libcrypto++9] action install (ceph-qa::ubuntu line 223) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[openmpi-common] action remove (ceph-qa::ubuntu line 234) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libuuid1] action install (ceph-qa::ubuntu line 238) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: Processing package[libfcgi] action install (ceph-qa::ubuntu line 239) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:30-07:00] INFO: package[libfcgi] is a virtual package, actually acting on package[libfcgi0ldbl] 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[btrfs-tools] action install (ceph-qa::ubuntu line 240) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libatomic-ops-dev] action install (ceph-qa::ubuntu line 243) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[git-core] action install (ceph-qa::ubuntu line 246) 2014-04-01 15:37:31,223.223 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[attr] action install (ceph-qa::ubuntu line 247) 2014-04-01 15:37:38,585.585 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:30-07:00] INFO: Processing package[dbench] action install (ceph-qa::ubuntu line 248) 2014-04-01 15:37:38,585.585 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[bonnie++] action install (ceph-qa::ubuntu line 249) 2014-04-01 15:37:38,586.586 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[iozone3] action install (ceph-qa::ubuntu line 250) 2014-04-01 15:37:38,586.586 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[tiobench] action install (ceph-qa::ubuntu line 252) 2014-04-01 15:37:38,587.587 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[ltp-kernel-test] action install (ceph-qa::ubuntu line 267) 2014-04-01 15:37:38,587.587 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[valgrind] action install (ceph-qa::ubuntu line 269) 2014-04-01 15:37:38,588.588 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[python-nose] action install (ceph-qa::ubuntu line 270) 2014-04-01 15:37:38,588.588 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[mpich2] action install (ceph-qa::ubuntu line 271) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libmpich2-3] action install (ceph-qa::ubuntu line 272) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libmpich2-dev] action install (ceph-qa::ubuntu line 273) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[ant] action install (ceph-qa::ubuntu line 274) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libtool] action install (ceph-qa::ubuntu line 277) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[automake] action install (ceph-qa::ubuntu line 278) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[gettext] action install (ceph-qa::ubuntu line 279) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[uuid-dev] action install (ceph-qa::ubuntu line 280) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libacl1-dev] action install (ceph-qa::ubuntu line 281) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[bc] action install (ceph-qa::ubuntu line 282) 2014-04-01 15:37:38,589.589 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[xfsdump] action install (ceph-qa::ubuntu line 283) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[dmapi] action install (ceph-qa::ubuntu line 284) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: package[dmapi] is a virtual package, actually acting on package[libdm0] 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[xfslibs-dev] action install (ceph-qa::ubuntu line 285) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libattr1-dev] action install (ceph-qa::ubuntu line 286) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[sysprof] action install (ceph-qa::ubuntu line 290) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[vim] action install (ceph-qa::ubuntu line 292) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[pdsh] action install (ceph-qa::ubuntu line 293) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing package[collectl] action install (ceph-qa::ubuntu line 294) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing service[collectl] action disable (ceph-qa::ubuntu line 295) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:31-07:00] INFO: Processing service[collectl] action stop (ceph-qa::ubuntu line 295) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[blktrace] action install (ceph-qa::ubuntu line 300) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[python-numpy] action install (ceph-qa::ubuntu line 301) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[python-matplotlib] action install (ceph-qa::ubuntu line 302) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[mencoder] action install (ceph-qa::ubuntu line 303) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[kvm] action install (ceph-qa::ubuntu line 309) 2014-04-01 15:37:38,590.590 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[genisoimage] action install (ceph-qa::ubuntu line 311) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[libjson-xs-perl] action install (ceph-qa::ubuntu line 314) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[xml-twig-tools] action install (ceph-qa::ubuntu line 316) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[default-jdk] action install (ceph-qa::ubuntu line 319) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[junit4] action install (ceph-qa::ubuntu line 320) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[tgt] action install (ceph-qa::ubuntu line 323) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[open-iscsi] action install (ceph-qa::ubuntu line 326) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[smartmontools] action install (ceph-qa::ubuntu line 328) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[nagios-nrpe-server] action install (ceph-qa::ubuntu line 329) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[cifs-utils] action install (ceph-qa::ubuntu line 332) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:32-07:00] INFO: Processing package[distcc] action install (ceph-qa::ubuntu line 335) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing file[/etc/grub.d/02_force_timeout] action create (ceph-qa::ubuntu line 351) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing package[ntp] action install (ceph-qa::ubuntu line 364) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing cookbook_file[/etc/ntp.conf] action create (ceph-qa::ubuntu line 366) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing service[ntp] action enable (ceph-qa::ubuntu line 374) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: service[ntp] enabled 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing service[ntp] action start (ceph-qa::ubuntu line 374) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user_xattr to root mount options in fstab] action run (ceph-qa::ubuntu line 378) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: execute[add user_xattr to root mount options in fstab] ran successfully 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[enable xattr for this boot] action run (ceph-qa::ubuntu line 387) 2014-04-01 15:37:38,591.591 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: execute[enable xattr for this boot] ran successfully 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[allow fuse mounts to be used by non-owners] action run (ceph-qa::ubuntu line 391) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: execute[allow fuse mounts to be used by non-owners] ran successfully 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user ubuntu to group fuse] action run (ceph-qa::ubuntu line 395) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: execute[add user ubuntu to group fuse] ran successfully 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing file[/etc/fuse.conf] action create (ceph-qa::ubuntu line 399) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user ubuntu to group kvm] action run (ceph-qa::ubuntu line 403) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: execute[add user ubuntu to group kvm] ran successfully 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing directory[/home/ubuntu/.ssh] action create (ceph-qa::ubuntu line 407) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing execute[set up ssh keys] action run (ceph-qa::ubuntu line 413) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: execute[set up ssh keys] ran successfully 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/usr/sbin/megacli] action create (ceph-qa::ubuntu line 425) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/usr/sbin/cli64] action create (ceph-qa::ubuntu line 431) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing directory[/usr/libexec/] action create (ceph-qa::ubuntu line 440) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/usr/libexec/raid.pl] action create (ceph-qa::ubuntu line 446) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/usr/libexec/smart.pl] action create (ceph-qa::ubuntu line 452) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/usr/libexec/diskusage.pl] action create (ceph-qa::ubuntu line 458) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing cookbook_file[/etc/ssh/ssh_config] action create (ceph-qa::ubuntu line 467) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing execute[add ubuntu to disk group] action run (ceph-qa::ubuntu line 474) 2014-04-01 15:37:38,592.592 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: execute[add ubuntu to disk group] ran successfully 2014-04-01 15:37:38,593.593 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing execute[enable kernel logging to console] action run (ceph-qa::ubuntu line 482) 2014-04-01 15:37:38,593.593 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: execute[enable kernel logging to console] ran successfully 2014-04-01 15:37:38,593.593 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:34-07:00] INFO: Processing execute[update-grub] action run (ceph-qa::ubuntu line 535) 2014-04-01 15:37:38,593.593 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: execute[update-grub] ran successfully 2014-04-01 15:37:38,593.593 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing cookbook_file[/etc/init/ttyS1.conf] action create (ceph-qa::ubuntu line 541) 2014-04-01 15:37:39,817.817 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[dbench] action install (ceph-qa::ubuntu line 248) 2014-04-01 15:37:39,818.818 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[bonnie++] action install (ceph-qa::ubuntu line 249) 2014-04-01 15:37:39,818.818 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[iozone3] action install (ceph-qa::ubuntu line 250) 2014-04-01 15:37:39,818.818 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[tiobench] action install (ceph-qa::ubuntu line 252) 2014-04-01 15:37:39,819.819 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[ltp-kernel-test] action install (ceph-qa::ubuntu line 267) 2014-04-01 15:37:39,819.819 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[valgrind] action install (ceph-qa::ubuntu line 269) 2014-04-01 15:37:39,819.819 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[python-nose] action install (ceph-qa::ubuntu line 270) 2014-04-01 15:37:39,820.820 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[mpich2] action install (ceph-qa::ubuntu line 271) 2014-04-01 15:37:39,820.820 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libmpich2-3] action install (ceph-qa::ubuntu line 272) 2014-04-01 15:37:39,820.820 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libmpich2-dev] action install (ceph-qa::ubuntu line 273) 2014-04-01 15:37:39,821.821 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[ant] action install (ceph-qa::ubuntu line 274) 2014-04-01 15:37:39,821.821 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libtool] action install (ceph-qa::ubuntu line 277) 2014-04-01 15:37:39,821.821 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[automake] action install (ceph-qa::ubuntu line 278) 2014-04-01 15:37:39,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[gettext] action install (ceph-qa::ubuntu line 279) 2014-04-01 15:37:39,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[uuid-dev] action install (ceph-qa::ubuntu line 280) 2014-04-01 15:37:39,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libacl1-dev] action install (ceph-qa::ubuntu line 281) 2014-04-01 15:37:39,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[bc] action install (ceph-qa::ubuntu line 282) 2014-04-01 15:37:39,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[xfsdump] action install (ceph-qa::ubuntu line 283) 2014-04-01 15:37:39,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[dmapi] action install (ceph-qa::ubuntu line 284) 2014-04-01 15:37:39,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: package[dmapi] is a virtual package, actually acting on package[libdm0] 2014-04-01 15:37:39,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[xfslibs-dev] action install (ceph-qa::ubuntu line 285) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[libattr1-dev] action install (ceph-qa::ubuntu line 286) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[sysprof] action install (ceph-qa::ubuntu line 290) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[vim] action install (ceph-qa::ubuntu line 292) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:31-07:00] INFO: Processing package[pdsh] action install (ceph-qa::ubuntu line 293) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[collectl] action install (ceph-qa::ubuntu line 294) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing service[collectl] action disable (ceph-qa::ubuntu line 295) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing service[collectl] action stop (ceph-qa::ubuntu line 295) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[blktrace] action install (ceph-qa::ubuntu line 300) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[python-numpy] action install (ceph-qa::ubuntu line 301) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[python-matplotlib] action install (ceph-qa::ubuntu line 302) 2014-04-01 15:37:39,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[mencoder] action install (ceph-qa::ubuntu line 303) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[kvm] action install (ceph-qa::ubuntu line 309) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[genisoimage] action install (ceph-qa::ubuntu line 311) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[libjson-xs-perl] action install (ceph-qa::ubuntu line 314) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[xml-twig-tools] action install (ceph-qa::ubuntu line 316) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[default-jdk] action install (ceph-qa::ubuntu line 319) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[junit4] action install (ceph-qa::ubuntu line 320) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[tgt] action install (ceph-qa::ubuntu line 323) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[open-iscsi] action install (ceph-qa::ubuntu line 326) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[smartmontools] action install (ceph-qa::ubuntu line 328) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:32-07:00] INFO: Processing package[nagios-nrpe-server] action install (ceph-qa::ubuntu line 329) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing package[cifs-utils] action install (ceph-qa::ubuntu line 332) 2014-04-01 15:37:39,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing package[distcc] action install (ceph-qa::ubuntu line 335) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing file[/etc/grub.d/02_force_timeout] action create (ceph-qa::ubuntu line 351) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing package[ntp] action install (ceph-qa::ubuntu line 364) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing cookbook_file[/etc/ntp.conf] action create (ceph-qa::ubuntu line 366) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing service[ntp] action enable (ceph-qa::ubuntu line 374) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: service[ntp] enabled 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing service[ntp] action start (ceph-qa::ubuntu line 374) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user_xattr to root mount options in fstab] action run (ceph-qa::ubuntu line 378) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: execute[add user_xattr to root mount options in fstab] ran successfully 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[enable xattr for this boot] action run (ceph-qa::ubuntu line 387) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: execute[enable xattr for this boot] ran successfully 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[allow fuse mounts to be used by non-owners] action run (ceph-qa::ubuntu line 391) 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: execute[allow fuse mounts to be used by non-owners] ran successfully 2014-04-01 15:37:39,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user ubuntu to group fuse] action run (ceph-qa::ubuntu line 395) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: execute[add user ubuntu to group fuse] ran successfully 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing file[/etc/fuse.conf] action create (ceph-qa::ubuntu line 399) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:33-07:00] INFO: Processing execute[add user ubuntu to group kvm] action run (ceph-qa::ubuntu line 403) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:34-07:00] INFO: execute[add user ubuntu to group kvm] ran successfully 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:34-07:00] INFO: Processing directory[/home/ubuntu/.ssh] action create (ceph-qa::ubuntu line 407) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:34-07:00] INFO: Processing execute[set up ssh keys] action run (ceph-qa::ubuntu line 413) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: execute[set up ssh keys] ran successfully 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/usr/sbin/megacli] action create (ceph-qa::ubuntu line 425) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/usr/sbin/cli64] action create (ceph-qa::ubuntu line 431) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing directory[/usr/libexec/] action create (ceph-qa::ubuntu line 440) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/usr/libexec/raid.pl] action create (ceph-qa::ubuntu line 446) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/usr/libexec/smart.pl] action create (ceph-qa::ubuntu line 452) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/usr/libexec/diskusage.pl] action create (ceph-qa::ubuntu line 458) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing cookbook_file[/etc/ssh/ssh_config] action create (ceph-qa::ubuntu line 467) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing execute[add ubuntu to disk group] action run (ceph-qa::ubuntu line 474) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: execute[add ubuntu to disk group] ran successfully 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing execute[enable kernel logging to console] action run (ceph-qa::ubuntu line 482) 2014-04-01 15:37:39,828.828 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: execute[enable kernel logging to console] ran successfully 2014-04-01 15:37:39,829.829 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:36-07:00] INFO: Processing execute[update-grub] action run (ceph-qa::ubuntu line 535) 2014-04-01 15:37:39,829.829 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: execute[update-grub] ran successfully 2014-04-01 15:37:39,829.829 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing cookbook_file[/etc/init/ttyS1.conf] action create (ceph-qa::ubuntu line 541) 2014-04-01 15:37:42,985.985 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing service[ttyS1] action enable (ceph-qa::ubuntu line 559) 2014-04-01 15:37:42,986.986 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing service[ttyS1] action start (ceph-qa::ubuntu line 559) 2014-04-01 15:37:42,986.986 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing package[nfs-kernel-server] action install (ceph-qa::ubuntu line 578) 2014-04-01 15:37:42,986.986 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing package[ipcalc] action install (ceph-qa::ubuntu line 581) 2014-04-01 15:37:42,987.987 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing execute[set up static IP in /etc/hosts] action run (ceph-qa::ubuntu line 585) 2014-04-01 15:37:42,987.987 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: execute[set up static IP in /etc/hosts] ran successfully 2014-04-01 15:37:42,987.987 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing execute[set up static IP and 10gig interface] action run (ceph-qa::ubuntu line 593) 2014-04-01 15:37:42,988.988 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: execute[set up static IP and 10gig interface] ran successfully 2014-04-01 15:37:42,988.988 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:38-07:00] INFO: Processing execute[Installing CPAN Amazon::S3] action run (ceph-qa::ubuntu line 645) 2014-04-01 15:37:42,988.988 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: execute[Installing CPAN Amazon::S3] ran successfully 2014-04-01 15:37:42,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing file[/etc/resolvconf/resolv.conf.d/base] action create (ceph-qa::ubuntu line 650) 2014-04-01 15:37:42,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing file[/etc/sudoers.d/90-nagios] action create (ceph-qa::ubuntu line 662) 2014-04-01 15:37:42,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing cookbook_file[/etc/modules] action create (ceph-qa::ubuntu line 672) 2014-04-01 15:37:42,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing cookbook_file[/etc/nagios/nrpe.cfg] action create (ceph-qa::ubuntu line 680) 2014-04-01 15:37:42,990.990 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing service[nagios-nrpe-server] action enable (ceph-qa::ubuntu line 688) 2014-04-01 15:37:42,990.990 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: service[nagios-nrpe-server] enabled 2014-04-01 15:37:42,990.990 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing service[nagios-nrpe-server] action start (ceph-qa::ubuntu line 688) 2014-04-01 15:37:42,991.991 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: service[nagios-nrpe-server] started 2014-04-01 15:37:42,991.991 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing file[/etc/default/nagios-nrpe-server] action create (ceph-qa::ubuntu line 693) 2014-04-01 15:37:42,991.991 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing execute[Restarting resolvdns] action run (ceph-qa::ubuntu line 702) 2014-04-01 15:37:42,991.991 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: execute[Restarting resolvdns] ran successfully 2014-04-01 15:37:42,992.992 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing execute[Enabling auto-fsck fix to prevent boot hangup] action run (ceph-qa::ubuntu line 708) 2014-04-01 15:37:42,992.992 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: execute[Enabling auto-fsck fix to prevent boot hangup] ran successfully 2014-04-01 15:37:42,992.992 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing bash[ssh_max_sessions] action run (ceph-qa::ubuntu line 714) 2014-04-01 15:37:42,992.992 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:41-07:00] INFO: Processing service[ssh] action restart (ceph-qa::ubuntu line 730) 2014-04-01 15:37:42,992.992 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:42-07:00] INFO: service[ssh] restarted 2014-04-01 15:37:42,993.993 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:42-07:00] INFO: Processing file[/ceph-qa-ready] action create (ceph-qa::ubuntu line 735) 2014-04-01 15:37:42,993.993 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:42-07:00] INFO: Chef Run complete in 22.178195624 seconds 2014-04-01 15:37:42,993.993 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:42-07:00] INFO: Running report handlers 2014-04-01 15:37:42,993.993 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:42-07:00] INFO: Report handlers complete 2014-04-01 15:37:43,013.013 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T06:37:19-07:00] INFO: Forking chef instance to converge... 2014-04-01 15:37:43,046.046 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cleanup 2014-04-01 15:37:43,047.047 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + rm -rf /tmp/solo-from-scratch.PCuqvIiujsq1 2014-04-01 15:37:43,940.940 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing service[ttyS1] action enable (ceph-qa::ubuntu line 559) 2014-04-01 15:37:43,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing service[ttyS1] action start (ceph-qa::ubuntu line 559) 2014-04-01 15:37:43,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing package[nfs-kernel-server] action install (ceph-qa::ubuntu line 578) 2014-04-01 15:37:43,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing package[ipcalc] action install (ceph-qa::ubuntu line 581) 2014-04-01 15:37:43,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing execute[set up static IP in /etc/hosts] action run (ceph-qa::ubuntu line 585) 2014-04-01 15:37:43,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: execute[set up static IP in /etc/hosts] ran successfully 2014-04-01 15:37:43,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing execute[set up static IP and 10gig interface] action run (ceph-qa::ubuntu line 593) 2014-04-01 15:37:43,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: execute[set up static IP and 10gig interface] ran successfully 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:39-07:00] INFO: Processing execute[Installing CPAN Amazon::S3] action run (ceph-qa::ubuntu line 645) 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: execute[Installing CPAN Amazon::S3] ran successfully 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing file[/etc/resolvconf/resolv.conf.d/base] action create (ceph-qa::ubuntu line 650) 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing file[/etc/sudoers.d/90-nagios] action create (ceph-qa::ubuntu line 662) 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing cookbook_file[/etc/modules] action create (ceph-qa::ubuntu line 672) 2014-04-01 15:37:43,943.943 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing cookbook_file[/etc/nagios/nrpe.cfg] action create (ceph-qa::ubuntu line 680) 2014-04-01 15:37:43,944.944 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing service[nagios-nrpe-server] action enable (ceph-qa::ubuntu line 688) 2014-04-01 15:37:43,944.944 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: service[nagios-nrpe-server] enabled 2014-04-01 15:37:43,944.944 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing service[nagios-nrpe-server] action start (ceph-qa::ubuntu line 688) 2014-04-01 15:37:43,944.944 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: service[nagios-nrpe-server] started 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing file[/etc/default/nagios-nrpe-server] action create (ceph-qa::ubuntu line 693) 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing execute[Restarting resolvdns] action run (ceph-qa::ubuntu line 702) 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: execute[Restarting resolvdns] ran successfully 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing execute[Enabling auto-fsck fix to prevent boot hangup] action run (ceph-qa::ubuntu line 708) 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: execute[Enabling auto-fsck fix to prevent boot hangup] ran successfully 2014-04-01 15:37:43,945.945 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing bash[ssh_max_sessions] action run (ceph-qa::ubuntu line 714) 2014-04-01 15:37:43,946.946 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:42-07:00] INFO: Processing service[ssh] action restart (ceph-qa::ubuntu line 730) 2014-04-01 15:37:43,946.946 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:43-07:00] INFO: service[ssh] restarted 2014-04-01 15:37:43,946.946 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:43-07:00] INFO: Processing file[/ceph-qa-ready] action create (ceph-qa::ubuntu line 735) 2014-04-01 15:37:43,946.946 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:43-07:00] INFO: Chef Run complete in 22.821317913 seconds 2014-04-01 15:37:43,946.946 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:43-07:00] INFO: Running report handlers 2014-04-01 15:37:43,947.947 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:43-07:00] INFO: Report handlers complete 2014-04-01 15:37:43,970.970 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T06:37:19-07:00] INFO: Forking chef instance to converge... 2014-04-01 15:37:44,002.002 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cleanup 2014-04-01 15:37:44,003.003 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + rm -rf /tmp/solo-from-scratch.L3KJTJOfGQkl 2014-04-01 15:37:44,009.009 INFO:teuthology.task.chef:Reconnecting after ceph-qa-chef run 2014-04-01 15:37:44,009.009 INFO:teuthology.misc:Re-opening connections... 2014-04-01 15:37:44,010.010 INFO:teuthology.misc:trying to connect to ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:37:45,702.702 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'echo online' 2014-04-01 15:37:46,024.024 INFO:teuthology.orchestra.run.out:[10.214.133.37]: online 2014-04-01 15:37:46,175.175 DEBUG:teuthology.misc:waited 2.16546106339 2014-04-01 15:37:47,174.174 INFO:teuthology.misc:trying to connect to ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:37:48,870.870 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'echo online' 2014-04-01 15:37:49,192.192 INFO:teuthology.orchestra.run.out:[10.214.133.38]: online 2014-04-01 15:37:49,345.345 DEBUG:teuthology.misc:waited 5.3351380825 2014-04-01 15:37:50,344.344 INFO:teuthology.run_tasks:Running task clock.check... 2014-04-01 15:37:50,345.345 INFO:teuthology.task.clock:Checking initial clock skew... 2014-04-01 15:37:50,346.346 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 15:37:50,705.705 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: remote local st poll reach delay offset disp 2014-04-01 15:37:50,706.706 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: ======================================================================= 2014-04-01 15:37:50,709.709 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: *clock3.dreamhos 10.214.133.37 1 16 77 0.00032 0.000842 0.02531 2014-04-01 15:37:50,709.709 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 15:37:51,077.077 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: remote local st poll reach delay offset disp 2014-04-01 15:37:51,077.077 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: ======================================================================= 2014-04-01 15:37:51,077.077 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: *clock3.dreamhos 10.214.133.38 1 16 177 0.00034 0.000954 0.03525 2014-04-01 15:37:51,188.188 INFO:teuthology.run_tasks:Running task install... 2014-04-01 15:37:51,189.189 DEBUG:teuthology.task.install:project ceph 2014-04-01 15:37:51,189.189 DEBUG:teuthology.task.install:config {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045'} 2014-04-01 15:37:51,189.189 INFO:teuthology.task.install:extra packages: [] 2014-04-01 15:37:51,190.190 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 15:37:51,554.554 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 15:37:51,554.554 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 15:37:51,554.554 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo apt-key list | grep Ceph' 2014-04-01 15:37:51,920.920 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 15:37:51,921.921 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo apt-key list | grep Ceph' 2014-04-01 15:37:52,051.051 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'arch' 2014-04-01 15:37:52,427.427 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'arch' 2014-04-01 15:37:52,673.673 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -is' 2014-04-01 15:37:53,066.066 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -is' 2014-04-01 15:37:53,256.256 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -rs' 2014-04-01 15:37:53,681.681 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -rs' 2014-04-01 15:37:53,801.801 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -sc' 2014-04-01 15:37:54,210.210 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -sc' 2014-04-01 15:37:54,350.350 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 15:37:54,351.351 INFO:teuthology.task.install:Installing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg, librados2, librados2-dbg, librbd1, librbd1-dbg on remote deb x86_64 2014-04-01 15:37:54,351.351 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'arch' 2014-04-01 15:37:54,760.760 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 15:37:54,760.760 INFO:teuthology.task.install:Installing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg, librados2, librados2-dbg, librbd1, librbd1-dbg on remote deb x86_64 2014-04-01 15:37:54,761.761 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'arch' 2014-04-01 15:37:55,013.013 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -is' 2014-04-01 15:37:55,389.389 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -is' 2014-04-01 15:37:55,578.578 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -rs' 2014-04-01 15:37:55,955.955 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -rs' 2014-04-01 15:37:56,078.078 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -sc' 2014-04-01 15:37:56,457.457 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -sc' 2014-04-01 15:37:56,577.577 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 15:37:56,578.578 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 15:37:56,959.959 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 15:37:56,960.960 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 15:37:57,082.082 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 15:37:57,083.083 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 2014-04-01 15:37:57,083.083 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045/version' 2014-04-01 15:37:57,476.476 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 15:37:57,476.476 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 2014-04-01 15:37:57,477.477 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045/version' 2014-04-01 15:37:57,591.591 INFO:teuthology.task.install:Package version is 0.78-459-ge672c52-1precise 2014-04-01 15:37:57,592.592 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'echo deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 precise main | sudo tee /etc/apt/sources.list.d/ceph.list' 2014-04-01 15:37:58,001.001 INFO:teuthology.task.install:Package version is 0.78-459-ge672c52-1precise 2014-04-01 15:37:58,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'echo deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 precise main | sudo tee /etc/apt/sources.list.d/ceph.list' 2014-04-01 15:37:58,115.115 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.78-459-ge672c52-1precise ceph-dbg=0.78-459-ge672c52-1precise ceph-mds=0.78-459-ge672c52-1precise ceph-mds-dbg=0.78-459-ge672c52-1precise ceph-common=0.78-459-ge672c52-1precise ceph-common-dbg=0.78-459-ge672c52-1precise ceph-fuse=0.78-459-ge672c52-1precise ceph-fuse-dbg=0.78-459-ge672c52-1precise ceph-test=0.78-459-ge672c52-1precise ceph-test-dbg=0.78-459-ge672c52-1precise radosgw=0.78-459-ge672c52-1precise radosgw-dbg=0.78-459-ge672c52-1precise python-ceph=0.78-459-ge672c52-1precise libcephfs1=0.78-459-ge672c52-1precise libcephfs1-dbg=0.78-459-ge672c52-1precise librados2=0.78-459-ge672c52-1precise librados2-dbg=0.78-459-ge672c52-1precise librbd1=0.78-459-ge672c52-1precise librbd1-dbg=0.78-459-ge672c52-1precise' 2014-04-01 15:37:58,554.554 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.78-459-ge672c52-1precise ceph-dbg=0.78-459-ge672c52-1precise ceph-mds=0.78-459-ge672c52-1precise ceph-mds-dbg=0.78-459-ge672c52-1precise ceph-common=0.78-459-ge672c52-1precise ceph-common-dbg=0.78-459-ge672c52-1precise ceph-fuse=0.78-459-ge672c52-1precise ceph-fuse-dbg=0.78-459-ge672c52-1precise ceph-test=0.78-459-ge672c52-1precise ceph-test-dbg=0.78-459-ge672c52-1precise radosgw=0.78-459-ge672c52-1precise radosgw-dbg=0.78-459-ge672c52-1precise python-ceph=0.78-459-ge672c52-1precise libcephfs1=0.78-459-ge672c52-1precise libcephfs1-dbg=0.78-459-ge672c52-1precise librados2=0.78-459-ge672c52-1precise librados2-dbg=0.78-459-ge672c52-1precise librbd1=0.78-459-ge672c52-1precise librbd1-dbg=0.78-459-ge672c52-1precise' 2014-04-01 15:38:58,836.836 DEBUG:teuthology.parallel:result is None 2014-04-01 15:38:59,680.680 DEBUG:teuthology.parallel:result is None 2014-04-01 15:38:59,681.681 INFO:teuthology.run_tasks:Running task ceph... 2014-04-01 15:38:59,686.686 INFO:teuthology.task.ceph:Making ceph log dir writeable by non-root... 2014-04-01 15:38:59,686.686 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod 777 /var/log/ceph' 2014-04-01 15:39:00,144.144 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod 777 /var/log/ceph' 2014-04-01 15:39:00,472.472 INFO:teuthology.task.ceph:Disabling ceph logrotate... 2014-04-01 15:39:00,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo rm -f -- /etc/logrotate.d/ceph' 2014-04-01 15:39:00,795.795 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo rm -f -- /etc/logrotate.d/ceph' 2014-04-01 15:39:01,129.129 INFO:teuthology.task.ceph:Creating extra log directories... 2014-04-01 15:39:01,129.129 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo install -d -m0755 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger' 2014-04-01 15:39:01,451.451 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo install -d -m0755 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger' 2014-04-01 15:39:01,782.782 INFO:teuthology.task.ceph:Shipping valgrind.supp... 2014-04-01 15:39:01,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/valgrind.supp' 2014-04-01 15:39:02,584.584 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/valgrind.supp' 2014-04-01 15:39:03,250.250 INFO:teuthology.task.ceph:Shipping 'daemon-helper'... 2014-04-01 15:39:03,251.251 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/daemon-helper' 2014-04-01 15:39:03,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/daemon-helper' 2014-04-01 15:39:04,412.412 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/daemon-helper' 2014-04-01 15:39:05,042.042 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/daemon-helper' 2014-04-01 15:39:05,574.574 INFO:teuthology.task.ceph:Shipping 'adjust-ulimits'... 2014-04-01 15:39:05,575.575 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/adjust-ulimits' 2014-04-01 15:39:06,239.239 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/adjust-ulimits' 2014-04-01 15:39:06,771.771 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/adjust-ulimits' 2014-04-01 15:39:07,439.439 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/adjust-ulimits' 2014-04-01 15:39:07,973.973 INFO:teuthology.task.ceph:Shipping 'kcon_most'... 2014-04-01 15:39:07,974.974 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/kcon_most' 2014-04-01 15:39:08,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/kcon_most' 2014-04-01 15:39:09,173.173 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/kcon_most' 2014-04-01 15:39:09,839.839 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/kcon_most' 2014-04-01 15:39:10,379.379 INFO:teuthology.task.ceph:Creating ceph cluster... 2014-04-01 15:39:10,379.379 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/data' 2014-04-01 15:39:10,922.922 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/data' 2014-04-01 15:39:11,454.454 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo install -d -m0777 -- /var/run/ceph' 2014-04-01 15:39:11,782.782 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo install -d -m0777 -- /var/run/ceph' 2014-04-01 15:39:12,114.114 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /scratch_devs' 2014-04-01 15:39:12,693.693 INFO:teuthology.orchestra.run.err:[10.214.133.37]: cat: /scratch_devs: No such file or directory 2014-04-01 15:39:12,694.694 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'ls /dev/[sv]d?' 2014-04-01 15:39:13,504.504 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 15:39:13,505.505 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda' 2014-04-01 15:39:13,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 15:39:13,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 15:39:13,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 4.2081e-05 s, 12.2 MB/s 2014-04-01 15:39:13,891.891 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sda' 2014-04-01 15:39:13,891.891 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:13,892.892 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 1236 Links: 1 Device type: 8,0 2014-04-01 15:39:13,892.892 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:13,892.892 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 06:35:49.233784858 -0700 2014-04-01 15:39:13,892.892 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 06:35:49.121784862 -0700 2014-04-01 15:39:13,893.893 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 06:35:49.121784862 -0700 2014-04-01 15:39:13,893.893 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 15:39:13,893.893 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2014-04-01 15:39:13,893.893 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb' 2014-04-01 15:39:14,462.462 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdb' 2014-04-01 15:39:14,462.462 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:14,463.463 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 12346 Links: 1 Device type: 8,10 2014-04-01 15:39:14,463.463 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:14,463.463 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 06:37:20.940143372 -0700 2014-04-01 15:39:14,464.464 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:14,464.464 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:14,464.464 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 15:39:14,472.472 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 15:39:14,472.472 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 15:39:14,473.473 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000610091 s, 839 kB/s 2014-04-01 15:39:14,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc' 2014-04-01 15:39:15,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdc' 2014-04-01 15:39:15,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:15,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 13364 Links: 1 Device type: 8,20 2014-04-01 15:39:15,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:15,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 06:37:20.952143732 -0700 2014-04-01 15:39:15,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:15,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:15,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 15:39:15,060.060 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 15:39:15,060.060 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 15:39:15,061.061 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000377598 s, 1.4 MB/s 2014-04-01 15:39:15,061.061 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd' 2014-04-01 15:39:15,604.604 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 15:39:15,604.604 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 15:39:15,605.605 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000355305 s, 1.4 MB/s 2014-04-01 15:39:15,605.605 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdd' 2014-04-01 15:39:15,605.605 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:15,605.605 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 10490 Links: 1 Device type: 8,30 2014-04-01 15:39:15,606.606 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:15,606.606 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 06:37:20.968144217 -0700 2014-04-01 15:39:15,606.606 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:15,606.606 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 06:35:47.521784918 -0700 2014-04-01 15:39:15,607.607 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 15:39:15,607.607 INFO:teuthology.task.ceph:fs option selected, checking for scratch devs 2014-04-01 15:39:15,607.607 INFO:teuthology.task.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 15:39:15,607.607 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "ls -l '/dev/disk/by-id/wwn-*'" 2014-04-01 15:39:16,143.143 INFO:teuthology.orchestra.run.err:[10.214.133.37]: ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2014-04-01 15:39:16,299.299 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2014-04-01 15:39:16,299.299 INFO:teuthology.task.ceph:dev map: {'1': '/dev/sdb', '0': '/dev/sdd', '2': '/dev/sdc'} 2014-04-01 15:39:16,300.300 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'cat -- /scratch_devs' 2014-04-01 15:39:16,916.916 INFO:teuthology.orchestra.run.err:[10.214.133.38]: cat: /scratch_devs: No such file or directory 2014-04-01 15:39:16,916.916 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'ls /dev/[sv]d?' 2014-04-01 15:39:17,607.607 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 15:39:17,607.607 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda' 2014-04-01 15:39:18,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sda' 2014-04-01 15:39:18,053.053 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:18,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13377 Links: 1 Device type: 8,0 2014-04-01 15:39:18,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:18,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 06:35:36.238622855 -0700 2014-04-01 15:39:18,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 06:35:36.170622857 -0700 2014-04-01 15:39:18,054.054 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 06:35:36.170622857 -0700 2014-04-01 15:39:18,055.055 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 15:39:18,055.055 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 15:39:18,055.055 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 15:39:18,055.055 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 3.986e-05 s, 12.8 MB/s 2014-04-01 15:39:18,055.055 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2014-04-01 15:39:18,055.055 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb' 2014-04-01 15:39:18,595.595 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdb' 2014-04-01 15:39:18,596.596 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:18,596.596 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13387 Links: 1 Device type: 8,10 2014-04-01 15:39:18,596.596 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:18,597.597 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 06:37:20.633903339 -0700 2014-04-01 15:39:18,597.597 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:18,597.597 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:18,597.597 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 15:39:18,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 15:39:18,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 15:39:18,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.000586565 s, 873 kB/s 2014-04-01 15:39:18,599.599 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc' 2014-04-01 15:39:19,135.135 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 15:39:19,136.136 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 15:39:19,136.136 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.00035785 s, 1.4 MB/s 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdc' 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13388 Links: 1 Device type: 8,20 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 06:37:20.645903518 -0700 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:19,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 15:39:19,137.137 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd' 2014-04-01 15:39:19,677.677 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdd' 2014-04-01 15:39:19,677.677 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 15:39:19,680.680 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13389 Links: 1 Device type: 8,30 2014-04-01 15:39:19,681.681 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 15:39:19,681.681 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 06:37:20.657903699 -0700 2014-04-01 15:39:19,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:19,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 06:35:35.594622878 -0700 2014-04-01 15:39:19,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 15:39:19,686.686 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 15:39:19,686.686 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 15:39:19,687.687 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.000370168 s, 1.4 MB/s 2014-04-01 15:39:19,687.687 INFO:teuthology.task.ceph:fs option selected, checking for scratch devs 2014-04-01 15:39:19,687.687 INFO:teuthology.task.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 15:39:19,688.688 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "ls -l '/dev/disk/by-id/wwn-*'" 2014-04-01 15:39:20,226.226 INFO:teuthology.orchestra.run.err:[10.214.133.38]: ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2014-04-01 15:39:20,381.381 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2014-04-01 15:39:20,382.382 INFO:teuthology.task.ceph:dev map: {'3': '/dev/sdd', '5': '/dev/sdc', '4': '/dev/sdb'} 2014-04-01 15:39:20,382.382 INFO:teuthology.task.ceph:Generating config... 2014-04-01 15:39:20,385.385 INFO:teuthology.task.ceph:[global] ms inject socket failures = 5000 2014-04-01 15:39:20,385.385 INFO:teuthology.task.ceph:[mon] debug mon = 20 2014-04-01 15:39:20,386.386 INFO:teuthology.task.ceph:[mon] debug paxos = 20 2014-04-01 15:39:20,386.386 INFO:teuthology.task.ceph:[mon] debug ms = 1 2014-04-01 15:39:20,386.386 INFO:teuthology.task.ceph:[osd] debug osd = 20 2014-04-01 15:39:20,387.387 INFO:teuthology.task.ceph:[osd] debug filestore = 20 2014-04-01 15:39:20,387.387 INFO:teuthology.task.ceph:[osd] debug journal = 20 2014-04-01 15:39:20,387.387 INFO:teuthology.task.ceph:[osd] debug ms = 1 2014-04-01 15:39:20,388.388 INFO:teuthology.task.ceph:Setting up mon.a... 2014-04-01 15:39:20,388.388 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring' 2014-04-01 15:39:20,843.843 INFO:teuthology.orchestra.run.out:[10.214.133.37]: creating /etc/ceph/ceph.keyring 2014-04-01 15:39:20,845.845 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring' 2014-04-01 15:39:21,328.328 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 15:39:21,812.812 DEBUG:teuthology.misc:Ceph mon addresses: [('a', '10.214.133.37:6789'), ('c', '10.214.133.37:6790'), ('b', '10.214.133.38:6789')] 2014-04-01 15:39:21,813.813 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --add a 10.214.133.37:6789 --add c 10.214.133.37:6790 --add b 10.214.133.38:6789 --print /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:22,326.326 INFO:teuthology.task.ceph:Writing ceph.conf for FSID a7571e41-653a-4300-8fe0-63e1a9732593... 2014-04-01 15:39:22,328.328 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2014-04-01 15:39:22,651.651 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2014-04-01 15:39:23,310.310 INFO:teuthology.task.ceph:Creating admin key on mon.a... 2014-04-01 15:39:23,311.311 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds allow /etc/ceph/ceph.keyring" 2014-04-01 15:39:23,860.860 INFO:teuthology.task.ceph:Copying monmap to all nodes... 2014-04-01 15:39:23,860.860 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /etc/ceph/ceph.keyring' 2014-04-01 15:39:24,488.488 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:25,165.165 INFO:teuthology.task.ceph:Sending monmap to node ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:39:25,166.166 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.keyring && sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 15:39:26,011.011 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:26,882.882 INFO:teuthology.task.ceph:Sending monmap to node ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:39:26,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.keyring && sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 15:39:27,557.557 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:28,391.391 INFO:teuthology.task.ceph:Setting up mon nodes... 2014-04-01 15:39:28,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage osdmaptool -c /etc/ceph/ceph.conf --clobber --createsimple 6 /home/ubuntu/cephtest/osdmap --pg_bits 2 --pgp_bits 4' 2014-04-01 15:39:28,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage osdmaptool -c /etc/ceph/ceph.conf --clobber --createsimple 6 /home/ubuntu/cephtest/osdmap --pg_bits 2 --pgp_bits 4' 2014-04-01 15:39:28,947.947 INFO:teuthology.orchestra.run.err:[10.214.133.38]: osdmaptool: osdmap file '/home/ubuntu/cephtest/osdmap' 2014-04-01 15:39:28,949.949 INFO:teuthology.orchestra.run.out:[10.214.133.38]: osdmaptool: writing epoch 1 to /home/ubuntu/cephtest/osdmap 2014-04-01 15:39:29,396.396 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osdmaptool: osdmap file '/home/ubuntu/cephtest/osdmap' 2014-04-01 15:39:29,397.397 INFO:teuthology.orchestra.run.out:[10.214.133.37]: osdmaptool: writing epoch 1 to /home/ubuntu/cephtest/osdmap 2014-04-01 15:39:29,397.397 INFO:teuthology.task.ceph:Setting up mds nodes... 2014-04-01 15:39:29,398.398 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring' 2014-04-01 15:39:29,748.748 INFO:teuthology.orchestra.run.out:[10.214.133.38]: creating /var/lib/ceph/mds/ceph-a/keyring 2014-04-01 15:39:29,749.749 INFO:teuthology.task.ceph_client:Setting up client nodes... 2014-04-01 15:39:29,749.749 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring' 2014-04-01 15:39:30,090.090 INFO:teuthology.orchestra.run.out:[10.214.133.37]: creating /etc/ceph/ceph.client.0.keyring 2014-04-01 15:39:30,098.098 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring' 2014-04-01 15:39:30,438.438 INFO:teuthology.orchestra.run.out:[10.214.133.38]: creating /etc/ceph/ceph.client.1.keyring 2014-04-01 15:39:30,446.446 INFO:teuthology.task.ceph:Running mkfs on osd nodes... 2014-04-01 15:39:30,447.447 INFO:teuthology.task.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@plana79.front.sepia.ceph.com'): {'1': '/dev/sdb', '0': '/dev/sdd', '2': '/dev/sdc'}, Remote(name='ubuntu@plana78.front.sepia.ceph.com'): {'3': '/dev/sdd', '5': '/dev/sdc', '4': '/dev/sdb'}} 2014-04-01 15:39:30,447.447 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-0' 2014-04-01 15:39:30,776.776 INFO:teuthology.task.ceph:{} 2014-04-01 15:39:30,777.777 INFO:teuthology.task.ceph:0 2014-04-01 15:39:30,777.777 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdd on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:39:30,777.777 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdd' 2014-04-01 15:39:31,311.311 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:39:31,329.329 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdd is entire device, not just one partition! 2014-04-01 15:39:31,329.329 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:39:31,329.329 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 15:39:31,329.329 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 15:39:31,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 15:39:31,331.331 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:39:31,331.331 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:39:31,331.331 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 15:39:31,331.331 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:34,913.913 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:35,734.734 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:35,737.737 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 15:39:38,130.130 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:38,130.130 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:38,131.131 INFO:teuthology.task.ceph:mount /dev/sdd on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:39:38,131.131 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdd /var/lib/ceph/osd/ceph-0' 2014-04-01 15:39:38,645.645 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-1' 2014-04-01 15:39:39,212.212 INFO:teuthology.task.ceph:{} 2014-04-01 15:39:39,213.213 INFO:teuthology.task.ceph:1 2014-04-01 15:39:39,213.213 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdb on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:39:39,214.214 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdb' 2014-04-01 15:39:39,785.785 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:39:39,802.802 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdb is entire device, not just one partition! 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:39:39,803.803 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:39:39,804.804 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 15:39:39,805.805 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 15:39:39,805.805 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:39:39,805.805 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:39:39,805.805 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 15:39:39,805.805 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:40,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:41,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:41,057.057 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 15:39:43,876.876 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:43,877.877 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:43,879.879 INFO:teuthology.task.ceph:mount /dev/sdb on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:39:43,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdb /var/lib/ceph/osd/ceph-1' 2014-04-01 15:39:44,388.388 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-2' 2014-04-01 15:39:44,855.855 INFO:teuthology.task.ceph:{} 2014-04-01 15:39:44,856.856 INFO:teuthology.task.ceph:2 2014-04-01 15:39:44,856.856 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdc on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 15:39:44,857.857 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdc' 2014-04-01 15:39:45,357.357 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:39:45,377.377 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdc is entire device, not just one partition! 2014-04-01 15:39:45,378.378 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:39:45,378.378 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 15:39:45,378.378 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 15:39:45,378.378 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 15:39:45,379.379 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:39:45,379.379 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 15:39:45,379.379 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:39:45,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 15:39:45,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 15:39:45,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 15:39:45,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:39:45,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 15:39:45,381.381 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 15:39:45,381.381 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:39:45,381.381 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:39:45,381.381 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 15:39:45,381.381 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:46,134.134 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:46,350.350 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:46,350.350 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 15:39:49,173.173 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:49,173.173 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 15:39:49,174.174 INFO:teuthology.task.ceph:mount /dev/sdc on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:39:49,174.174 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdc /var/lib/ceph/osd/ceph-2' 2014-04-01 15:39:49,687.687 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:50,534.534 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:50.448492 7fe656513780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:51,027.027 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:50.942674 7fe656513780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:51,042.042 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:50.957927 7fe656513780 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:39:51,532.532 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:51.446276 7fe656513780 -1 created object store /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:39:51,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:51.446324 7fe656513780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2014-04-01 15:39:51,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:51.446401 7fe656513780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2014-04-01 15:39:51,534.534 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:52,333.333 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:52.248224 7f511c525780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:52,827.827 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:52.742240 7f511c525780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:52,843.843 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:52.758460 7f511c525780 -1 filestore(/var/lib/ceph/osd/ceph-1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:39:53,323.323 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:53.238646 7f511c525780 -1 created object store /var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:39:53,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:53.238694 7f511c525780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2014-04-01 15:39:53,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:53.238780 7f511c525780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2014-04-01 15:39:53,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:39:54,073.073 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:53.988397 7faf4a490780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:54,501.501 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:54.415731 7faf4a490780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:39:54,518.518 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:54.433692 7faf4a490780 -1 filestore(/var/lib/ceph/osd/ceph-2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:39:55,004.004 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:54.919330 7faf4a490780 -1 created object store /var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:39:55,005.005 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:54.919376 7faf4a490780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2014-04-01 15:39:55,005.005 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:39:54.919451 7faf4a490780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2014-04-01 15:39:55,007.007 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-3' 2014-04-01 15:39:55,336.336 INFO:teuthology.task.ceph:{} 2014-04-01 15:39:55,337.337 INFO:teuthology.task.ceph:3 2014-04-01 15:39:55,337.337 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdd on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:39:55,338.338 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdd' 2014-04-01 15:39:55,853.853 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdd is entire device, not just one partition! 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 15:39:55,869.869 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 15:39:55,870.870 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:39:56,666.666 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:56,840.840 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:56,840.840 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 15:39:59,653.653 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:39:59,654.654 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:39:59,656.656 INFO:teuthology.task.ceph:mount /dev/sdd on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:39:59,657.657 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdd /var/lib/ceph/osd/ceph-3' 2014-04-01 15:40:00,162.162 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-4' 2014-04-01 15:40:00,629.629 INFO:teuthology.task.ceph:{} 2014-04-01 15:40:00,629.629 INFO:teuthology.task.ceph:4 2014-04-01 15:40:00,630.630 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdb on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:40:00,630.630 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdb' 2014-04-01 15:40:01,129.129 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:40:01,148.148 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdb is entire device, not just one partition! 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 15:40:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:40:01,150.150 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:40:01,150.150 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 15:40:01,150.150 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:40:01,841.841 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:02,151.151 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:02,152.152 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 15:40:04,927.927 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:04,927.927 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:40:04,928.928 INFO:teuthology.task.ceph:mount /dev/sdb on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:40:04,928.928 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdb /var/lib/ceph/osd/ceph-4' 2014-04-01 15:40:05,436.436 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-5' 2014-04-01 15:40:05,897.897 INFO:teuthology.task.ceph:{} 2014-04-01 15:40:05,897.897 INFO:teuthology.task.ceph:5 2014-04-01 15:40:05,897.897 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdc on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 15:40:05,897.897 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdc' 2014-04-01 15:40:06,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdc is entire device, not just one partition! 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 15:40:06,417.417 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 15:40:06,418.418 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 15:40:06,418.418 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 15:40:06,418.418 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 15:40:06,418.418 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 15:40:06,418.418 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:40:07,066.066 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:07,337.337 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:07,338.338 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 15:40:10,038.038 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: 0/3727 1/3727 2/3727 3/3727 4/3727 5/3727 6/3727 7/3727 8/3727 9/3727 10/3727 11/3727 12/3727 13/3727 14/3727 15/3727 16/3727 17/3727 18/3727 19/3727 20/3727 21/3727 22/3727 23/3727 24/3727 25/3727 26/3727 27/3727 28/3727 29/3727 30/3727 31/3727 32/3727 33/3727 34/3727 35/3727 36/3727 37/3727 38/3727 39/3727 40/3727 41/3727 42/3727 43/3727 44/3727 45/3727 46/3727 47/3727 48/3727 49/3727 50/3727 51/3727 52/3727 53/3727 54/3727 55/3727 56/3727 57/3727 58/3727 59/3727 60/3727 61/3727 62/3727 63/3727 64/3727 65/3727 66/3727 67/3727 68/3727 69/3727 70/3727 71/3727 72/3727 73/3727 74/3727 75/3727 76/3727 77/3727 78/3727 79/3727 80/3727 81/3727 82/3727 83/3727 84/3727 85/3727 86/3727 87/3727 88/3727 89/3727 90/3727 91/3727 92/3727 93/3727 94/3727 95/3727 96/3727 97/3727 98/3727 99/3727 100/3727 101/3727 102/3727 103/3727 104/3727 105/3727 106/3727 107/3727 108/3727 109/3727 110/3727 111/3727 112/3727 113/3727 114/3727 115/3727 116/3727 117/3727 118/3727 119/3727 120/3727 121/3727 122/3727 123/3727 124/3727 125/3727 126/3727 127/3727 128/3727 129/3727 130/3727 131/3727 132/3727 133/3727 134/3727 135/3727 136/3727 137/3727 138/3727 139/3727 140/3727 141/3727 142/3727 143/3727 144/3727 145/3727 146/3727 147/3727 148/3727 149/3727 150/3727 151/3727 152/3727 153/3727 154/3727 155/3727 156/3727 157/3727 158/3727 159/3727 160/3727 161/3727 162/3727 163/3727 164/3727 165/3727 166/3727 167/3727 168/3727 169/3727 170/3727 171/3727 172/3727 173/3727 174/3727 175/3727 176/3727 177/3727 178/3727 179/3727 180/3727 181/3727 182/3727 183/3727 184/3727 185/3727 186/3727 187/3727 188/3727 189/3727 190/3727 191/3727 192/3727 193/3727 194/3727 195/3727 196/3727 197/3727 198/3727 199/3727 200/3727 201/3727 202/3727 203/3727 204/3727 205/3727 206/3727 207/3727 208/3727 209/3727 210/3727 211/3727 212/3727 213/3727 214/3727 215/3727 216/3727 217/3727 218/3727 219/3727 220/3727 221/3727 222/3727 223/3727 224/3727 225/3727 226/3727 227/3727 228/3727 229/3727 230/3727 231/3727 232/3727 233/3727 234/3727 235/3727 236/3727 237/3727 238/3727 239/3727 240/3727 241/3727 242/3727 243/3727 244/3727 245/3727 246/3727 247/3727 248/3727 249/3727 250/3727 251/3727 252/3727 253/3727 254/3727 255/3727 256/3727 257/3727 258/3727 259/3727 260/3727 261/3727 262/3727 263/3727 264/3727 265/3727 266/3727 267/3727 268/3727 269/3727 270/3727 271/3727 272/3727 273/3727 274/3727 275/3727 276/3727 277/3727 278/3727 279/3727 280/3727 281/3727 282/3727 283/3727 284/3727 285/3727 286/3727 287/3727 288/3727 289/3727 290/3727 291/3727 292/3727 293/3727 294/3727 295/3727 296/3727 297/3727 298/3727 299/3727 300/3727 301/3727 302/3727 303/3727 304/3727 305/3727 306/3727 307/3727 308/3727 309/3727 310/3727 311/3727 312/3727 313/3727 314/3727 315/3727 316/3727 317/3727 318/3727 319/3727 320/3727 321/3727 322/3727 323/3727 324/3727 325/3727 326/3727 327/3727 328/3727 329/3727 330/3727 331/3727 332/3727 333/3727 334/3727 335/3727 336/3727 337/3727 338/3727 339/3727 340/3727 341/3727 342/3727 343/3727 344/3727 345/3727 346/3727 347/3727 348/3727 349/3727 350/3727 351/3727 352/3727 353/3727 354/3727 355/3727 356/3727 357/3727 358/3727 359/3727 360/3727 361/3727 362/3727 363/3727 364/3727 365/3727 366/3727 367/3727 368/3727 369/3727 370/3727 371/3727 372/3727 373/3727 374/3727 375/3727 376/3727 377/3727 378/3727 379/3727 380/3727 381/3727 382/3727 383/3727 384/3727 385/3727 386/3727 387/3727 388/3727 389/3727 390/3727 391/3727 392/3727 393/3727 394/3727 395/3727 396/3727 397/3727 398/3727 399/3727 400/3727 401/3727 402/3727 403/3727 404/3727 405/3727 406/3727 407/3727 408/3727 409/3727 410/3727 411/3727 412/3727 413/3727 414/3727 415/3727 416/3727 417/3727 418/3727 419/3727 420/3727 421/3727 422/3727 423/3727 424/3727 425/3727 426/3727 427/3727 428/3727 429/3727 430/3727 431/3727 432/3727 433/3727 434/3727 435/3727 436/3727 437/3727 438/3727 439/3727 440/3727 441/3727 442/3727 443/3727 444/3727 445/3727 446/3727 447/3727 448/3727 449/3727 450/3727 451/3727 452/3727 453/3727 454/3727 455/3727 456/3727 457/3727 458/3727 459/3727 460/3727 461/3727 462/3727 463/3727 464/3727 465/3727 466/3727 467/3727 468/3727 469/3727 470/3727 471/3727 472/3727 473/3727 474/3727 475/3727 476/3727 477/3727 478/3727 479/3727 480/3727 481/3727 482/3727 483/3727 484/3727 485/3727 486/3727 487/3727 488/3727 489/3727 490/3727 491/3727 492/3727 493/3727 494/3727 495/3727 496/3727 497/3727 498/3727 499/3727 500/3727 501/3727 502/3727 503/3727 504/3727 505/3727 506/3727 507/3727 508/3727 509/3727 510/3727 511/3727 512/3727 513/3727 514/3727 515/3727 516/3727 517/3727 518/3727 519/3727 520/3727 521/3727 522/3727 523/3727 524/3727 525/3727 526/3727 527/3727 528/3727 529/3727 530/3727 531/3727 532/3727 533/3727 534/3727 535/3727 536/3727 537/3727 538/3727 539/3727 540/3727 541/3727 542/3727 543/3727 544/3727 545/3727 546/3727 547/3727 548/3727 549/3727 550/3727 551/3727 552/3727 553/3727 554/3727 555/3727 556/3727 557/3727 558/3727 559/3727 560/3727 561/3727 562/3727 563/3727 564/3727 565/3727 566/3727 567/3727 568/3727 569/3727 570/3727 571/3727 572/3727 573/3727 574/3727 575/3727 576/3727 577/3727 578/3727 579/3727 580/3727 581/3727 582/3727 583/3727 584/3727 585/3727 586/3727 587/3727 588/3727 589/3727 590/3727 591/3727 592/3727 593/3727 594/3727 595/3727 596/3727 597/3727 598/3727 599/3727 600/3727 601/3727 602/3727 603/3727 604/3727 605/3727 606/3727 607/3727 608/3727 609/3727 610/3727 611/3727 612/3727 613/3727 614/3727 615/3727 616/3727 617/3727 618/3727 619/3727 620/3727 621/3727 622/3727 623/3727 624/3727 625/3727 626/3727 627/3727 628/3727 629/3727 630/3727 631/3727 632/3727 633/3727 634/3727 635/3727 636/3727 637/3727 638/3727 639/3727 640/3727 641/3727 642/3727 643/3727 644/3727 645/3727 646/3727 647/3727 648/3727 649/3727 650/3727 651/3727 652/3727 653/3727 654/3727 655/3727 656/3727 657/3727 658/3727 659/3727 660/3727 661/3727 662/3727 663/3727 664/3727 665/3727 666/3727 667/3727 668/3727 669/3727 670/3727 671/3727 672/3727 673/3727 674/3727 675/3727 676/3727 677/3727 678/3727 679/3727 680/3727 681/3727 682/3727 683/3727 684/3727 685/3727 686/3727 687/3727 688/3727 689/3727 690/3727 691/3727 692/3727 693/3727 694/3727 695/3727 696/3727 697/3727 698/3727 699/3727 700/3727 701/3727 702/3727 703/3727 704/3727 705/3727 706/3727 707/3727 708/3727 709/3727 710/3727 711/3727 712/3727 713/3727 714/3727 715/3727 716/3727 717/3727 718/3727 719/3727 720/3727 721/3727 722/3727 723/3727 724/3727 725/3727 726/3727 727/3727 728/3727 729/3727 730/3727 731/3727 732/3727 733/3727 734/3727 735/3727 736/3727 737/3727 738/3727 739/3727 740/3727 741/3727 742/3727 743/3727 744/3727 745/3727 746/3727 747/3727 748/3727 749/3727 750/3727 751/3727 752/3727 753/3727 754/3727 755/3727 756/3727 757/3727 758/3727 759/3727 760/3727 761/3727 762/3727 763/3727 764/3727 765/3727 766/3727 767/3727 768/3727 769/3727 770/3727 771/3727 772/3727 773/3727 774/3727 775/3727 776/3727 777/3727 778/3727 779/3727 780/3727 781/3727 782/3727 783/3727 784/3727 785/3727 786/3727 787/3727 788/3727 789/3727 790/3727 791/3727 792/3727 793/3727 794/3727 795/3727 796/3727 797/3727 798/3727 799/3727 800/3727 801/3727 802/3727 803/3727 804/3727 805/3727 806/3727 807/3727 808/3727 809/3727 810/3727 811/3727 812/3727 813/3727 814/3727 815/3727 816/3727 817/3727 818/3727 819/3727 820/3727 821/3727 822/3727 823/3727 824/3727 825/3727 826/3727 827/3727 828/3727 829/3727 830/3727 831/3727 832/3727 833/3727 834/3727 835/3727 836/3727 837/3727 838/3727 839/3727 840/3727 841/3727 842/3727 843/3727 844/3727 845/3727 846/3727 847/3727 848/3727 849/3727 850/3727 851/3727 852/3727 853/3727 854/3727 855/3727 856/3727 857/3727 858/3727 859/3727 860/3727 861/3727 862/3727 863/3727 864/3727 865/3727 866/3727 867/3727 868/3727 869/3727 870/3727 871/3727 872/3727 873/3727 874/3727 875/3727 876/3727 877/3727 878/3727 879/3727 880/3727 881/3727 882/3727 883/3727 884/3727 885/3727 886/3727 887/3727 888/3727 889/3727 890/3727 891/3727 892/3727 893/3727 894/3727 895/3727 896/3727 897/3727 898/3727 899/3727 900/3727 901/3727 902/3727 903/3727 904/3727 905/3727 906/3727 907/3727 908/3727 909/3727 910/3727 911/3727 912/3727 913/3727 914/3727 915/3727 916/3727 917/3727 918/3727 919/3727 920/3727 921/3727 922/3727 923/3727 924/3727 925/3727 926/3727 927/3727 928/3727 929/3727 930/3727 931/3727 932/3727 933/3727 934/3727 935/3727 936/3727 937/3727 938/3727 939/3727 940/3727 941/3727 942/3727 943/3727 944/3727 945/3727 946/3727 947/3727 948/3727 949/3727 950/3727 951/3727 952/3727 953/3727 954/3727 955/3727 956/3727 957/3727 958/3727 959/3727 960/3727 961/3727 962/3727 963/3727 964/3727 965/3727 966/3727 967/3727 968/3727 969/3727 970/3727 971/3727 972/3727 973/3727 974/3727 975/3727 976/3727 977/3727 978/3727 979/3727 980/3727 981/3727 982/3727 983/3727 984/3727 985/3727 986/3727 987/3727 988/3727 989/3727 990/3727 991/3727 992/3727 993/3727 994/3727 995/3727 996/3727 997/3727 998/3727 999/37271000/37271001/37271002/37271003/37271004/37271005/37271006/37271007/37271008/37271009/37271010/37271011/37271012/37271013/37271014/37271015/37271016/37271017/37271018/37271019/37271020/37271021/37271022/37271023/37271024/37271025/37271026/37271027/37271028/37271029/37271030/37271031/37271032/37271033/37271034/37271035/37271036/37271037/37271038/37271039/37271040/37271041/37271042/37271043/37271044/37271045/37271046/37271047/37271048/37271049/37271050/37271051/37271052/37271053/37271054/37271055/37271056/37271057/37271058/37271059/37271060/37271061/37271062/37271063/37271064/37271065/37271066/37271067/37271068/37271069/37271070/37271071/37271072/37271073/37271074/37271075/37271076/37271077/37271078/37271079/37271080/37271081/37271082/37271083/37271084/37271085/37271086/37271087/37271088/37271089/37271090/37271091/37271092/37271093/37271094/37271095/37271096/37271097/37271098/37271099/37271100/37271101/37271102/37271103/37271104/37271105/37271106/37271107/37271108/37271109/37271110/37271111/37271112/37271113/37271114/37271115/37271116/37271117/37271118/37271119/37271120/37271121/37271122/37271123/37271124/37271125/37271126/37271127/37271128/37271129/37271130/37271131/37271132/37271133/37271134/37271135/37271136/37271137/37271138/37271139/37271140/37271141/37271142/37271143/37271144/37271145/37271146/37271147/37271148/37271149/37271150/37271151/37271152/37271153/37271154/37271155/37271156/37271157/37271158/37271159/37271160/37271161/37271162/37271163/37271164/37271165/37271166/37271167/37271168/37271169/37271170/37271171/37271172/37271173/37271174/37271175/37271176/37271177/37271178/37271179/37271180/37271181/37271182/37271183/37271184/37271185/37271186/37271187/37271188/37271189/37271190/37271191/37271192/37271193/37271194/37271195/37271196/37271197/37271198/37271199/37271200/37271201/37271202/37271203/37271204/37271205/37271206/37271207/37271208/37271209/37271210/37271211/37271212/37271213/37271214/37271215/37271216/37271217/37271218/37271219/37271220/37271221/37271222/37271223/37271224/37271225/37271226/37271227/37271228/37271229/37271230/37271231/37271232/37271233/37271234/37271235/37271236/37271237/37271238/37271239/37271240/37271241/37271242/37271243/37271244/37271245/37271246/37271247/37271248/37271249/37271250/37271251/37271252/37271253/37271254/37271255/37271256/37271257/37271258/37271259/37271260/37271261/37271262/37271263/37271264/37271265/37271266/37271267/37271268/37271269/37271270/37271271/37271272/37271273/37271274/37271275/37271276/37271277/37271278/37271279/37271280/37271281/37271282/37271283/37271284/37271285/37271286/37271287/37271288/37271289/37271290/37271291/37271292/37271293/37271294/37271295/37271296/37271297/37271298/37271299/37271300/37271301/37271302/37271303/37271304/37271305/37271306/37271307/37271308/37271309/37271310/37271311/37271312/37271313/37271314/37271315/37271316/37271317/37271318/37271319/37271320/37271321/37271322/37271323/37271324/37271325/37271326/37271327/37271328/37271329/37271330/37271331/37271332/37271333/37271334/37271335/37271336/37271337/37271338/37271339/37271340/37271341/37271342/37271343/37271344/37271345/37271346/37271347/37271348/37271349/37271350/37271351/37271352/37271353/37271354/37271355/37271356/37271357/37271358/37271359/37271360/37271361/37271362/37271363/37271364/37271365/37271366/37271367/37271368/37271369/37271370/37271371/37271372/37271373/37271374/37271375/37271376/37271377/37271378/37271379/37271380/37271381/37271382/37271383/37271384/37271385/37271386/37271387/37271388/37271389/37271390/37271391/37271392/37271393/37271394/37271395/37271396/37271397/37271398/37271399/37271400/37271401/37271402/37271403/37271404/37271405/37271406/37271407/37271408/37271409/37271410/37271411/37271412/37271413/37271414/37271415/37271416/37271417/37271418/37271419/37271420/37271421/37271422/37271423/37271424/37271425/37271426/37271427/37271428/37271429/37271430/37271431/37271432/37271433/37271434/37271435/37271436/37271437/37271438/37271439/37271440/37271441/37271442/37271443/37271444/37271445/37271446/37271447/37271448/37271449/37271450/37271451/37271452/37271453/37271454/37271455/37271456/37271457/37271458/37271459/37271460/37271461/37271462/37271463/37271464/37271465/37271466/37271467/37271468/37271469/37271470/37271471/37271472/37271473/37271474/37271475/37271476/37271477/37271478/37271479/37271480/37271481/37271482/37271483/37271484/37271485/37271486/37271487/37271488/37271489/37271490/37271491/37271492/37271493/37271494/37271495/37271496/37271497/37271498/37271499/37271500/37271501/37271502/37271503/37271504/37271505/37271506/37271507/37271508/37271509/37271510/37271511/37271512/37271513/37271514/37271515/37271516/37271517/37271518/37271519/37271520/37271521/37271522/37271523/37271524/37271525/37271526/37271527/37271528/37271529/37271530/37271531/37271532/37271533/37271534/37271535/37271536/37271537/37271538/37271539/37271540/37271541/37271542/37271543/37271544/37271545/37271546/37271547/37271548/37271549/37271550/37271551/37271552/37271553/37271554/37271555/37271556/37271557/37271558/37271559/37271560/37271561/37271562/37271563/37271564/37271565/37271566/37271567/37271568/37271569/37271570/37271571/37271572/37271573/37271574/37271575/37271576/37271577/37271578/37271579/37271580/37271581/37271582/37271583/37271584/37271585/37271586/37271587/37271588/37271589/37271590/37271591/37271592/37271593/37271594/37271595/37271596/37271597/37271598/37271599/37271600/37271601/37271602/37271603/37271604/37271605/37271606/37271607/37271608/37271609/37271610/37271611/37271612/37271613/37271614/37271615/37271616/37271617/37271618/37271619/37271620/37271621/37271622/37271623/37271624/37271625/37271626/37271627/37271628/37271629/37271630/37271631/37271632/37271633/37271634/37271635/37271636/37271637/37271638/37271639/37271640/37271641/37271642/37271643/37271644/37271645/37271646/37271647/37271648/37271649/37271650/37271651/37271652/37271653/37271654/37271655/37271656/37271657/37271658/37271659/37271660/37271661/37271662/37271663/37271664/37271665/37271666/37271667/37271668/37271669/37271670/37271671/37271672/37271673/37271674/37271675/37271676/37271677/37271678/37271679/37271680/37271681/37271682/37271683/37271684/37271685/37271686/37271687/37271688/37271689/37271690/37271691/37271692/37271693/37271694/37271695/37271696/37271697/37271698/37271699/37271700/37271701/37271702/37271703/37271704/37271705/37271706/37271707/37271708/37271709/37271710/37271711/37271712/37271713/37271714/37271715/37271716/37271717/37271718/37271719/37271720/37271721/37271722/37271723/37271724/37271725/37271726/37271727/37271728/37271729/37271730/37271731/37271732/37271733/37271734/37271735/37271736/37271737/37271738/37271739/37271740/37271741/37271742/37271743/37271744/37271745/37271746/37271747/37271748/37271749/37271750/37271751/37271752/37271753/37271754/37271755/37271756/37271757/37271758/37271759/37271760/37271761/37271762/37271763/37271764/37271765/37271766/37271767/37271768/37271769/37271770/37271771/37271772/37271773/37271774/37271775/37271776/37271777/37271778/37271779/37271780/37271781/37271782/37271783/37271784/37271785/37271786/37271787/37271788/37271789/37271790/37271791/37271792/37271793/37271794/37271795/37271796/37271797/37271798/37271799/37271800/37271801/37271802/37271803/37271804/37271805/37271806/37271807/37271808/37271809/37271810/37271811/37271812/37271813/37271814/37271815/37271816/37271817/37271818/37271819/37271820/37271821/37271822/37271823/37271824/37271825/37271826/37271827/37271828/37271829/37271830/37271831/37271832/37271833/37271834/37271835/37271836/37271837/37271838/37271839/37271840/37271841/37271842/37271843/37271844/37271845/37271846/37271847/37271848/37271849/37271850/37271851/37271852/37271853/37271854/37271855/37271856/37271857/37271858/37271859/37271860/37271861/37271862/37271863/37271864/37271865/37271866/37271867/37271868/37271869/37271870/37271871/37271872/37271873/37271874/37271875/37271876/37271877/37271878/37271879/37271880/37271881/37271882/37271883/37271884/37271885/37271886/37271887/37271888/37271889/37271890/37271891/37271892/37271893/37271894/37271895/37271896/37271897/37271898/37271899/37271900/37271901/37271902/37271903/37271904/37271905/37271906/37271907/37271908/37271909/37271910/37271911/37271912/37271913/37271914/37271915/37271916/37271917/37271918/37271919/37271920/37271921/37271922/37271923/37271924/37271925/37271926/37271927/37271928/37271929/37271930/37271931/37271932/37271933/37271934/37271935/37271936/37271937/37271938/37271939/37271940/37271941/37271942/37271943/37271944/37271945/37271946/37271947/37271948/37271949/37271950/37271951/37271952/37271953/37271954/37271955/37271956/37271957/37271958/37271959/37271960/37271961/37271962/37271963/37271964/37271965/37271966/37271967/37271968/37271969/37271970/37271971/37271972/37271973/37271974/37271975/37271976/37271977/37271978/37271979/37271980/37271981/37271982/37271983/37271984/37271985/37271986/37271987/37271988/37271989/37271990/37271991/37271992/37271993/37271994/37271995/37271996/37271997/37271998/37271999/37272000/37272001/37272002/37272003/37272004/37272005/37272006/37272007/37272008/37272009/37272010/37272011/37272012/37272013/37272014/37272015/37272016/37272017/37272018/37272019/37272020/37272021/37272022/37272023/37272024/37272025/37272026/37272027/37272028/37272029/37272030/37272031/37272032/37272033/37272034/37272035/37272036/37272037/37272038/37272039/37272040/37272041/37272042/37272043/37272044/37272045/37272046/37272047/37272048/37272049/37272050/37272051/37272052/37272053/37272054/37272055/37272056/37272057/37272058/37272059/37272060/37272061/37272062/37272063/37272064/37272065/37272066/37272067/37272068/37272069/37272070/37272071/37272072/37272073/37272074/37272075/37272076/37272077/37272078/37272079/37272080/37272081/37272082/37272083/37272084/37272085/37272086/37272087/37272088/37272089/37272090/37272091/37272092/37272093/37272094/37272095/37272096/37272097/37272098/37272099/37272100/37272101/37272102/37272103/37272104/37272105/37272106/37272107/37272108/37272109/37272110/37272111/37272112/37272113/37272114/37272115/37272116/37272117/37272118/37272119/37272120/37272121/37272122/37272123/37272124/37272125/37272126/37272127/37272128/37272129/37272130/37272131/37272132/37272133/37272134/37272135/37272136/37272137/37272138/37272139/37272140/37272141/37272142/37272143/37272144/37272145/37272146/37272147/37272148/37272149/37272150/37272151/37272152/37272153/37272154/37272155/37272156/37272157/37272158/37272159/37272160/37272161/37272162/37272163/37272164/37272165/37272166/37272167/37272168/37272169/37272170/37272171/37272172/37272173/37272174/37272175/37272176/37272177/37272178/37272179/37272180/37272181/37272182/37272183/37272184/37272185/37272186/37272187/37272188/37272189/37272190/37272191/37272192/37272193/37272194/37272195/37272196/37272197/37272198/37272199/37272200/37272201/37272202/37272203/37272204/37272205/37272206/37272207/37272208/37272209/37272210/37272211/37272212/37272213/37272214/37272215/37272216/37272217/37272218/37272219/37272220/37272221/37272222/37272223/37272224/37272225/37272226/37272227/37272228/37272229/37272230/37272231/37272232/37272233/37272234/37272235/37272236/37272237/37272238/37272239/37272240/37272241/37272242/37272243/37272244/37272245/37272246/37272247/37272248/37272249/37272250/37272251/37272252/37272253/37272254/37272255/37272256/37272257/37272258/37272259/37272260/37272261/37272262/37272263/37272264/37272265/37272266/37272267/37272268/37272269/37272270/37272271/37272272/37272273/37272274/37272275/37272276/37272277/37272278/37272279/37272280/37272281/37272282/37272283/37272284/37272285/37272286/37272287/37272288/37272289/37272290/37272291/37272292/37272293/37272294/37272295/37272296/37272297/37272298/37272299/37272300/37272301/37272302/37272303/37272304/37272305/37272306/37272307/37272308/37272309/37272310/37272311/37272312/37272313/37272314/37272315/37272316/37272317/37272318/37272319/37272320/37272321/37272322/37272323/37272324/37272325/37272326/37272327/37272328/37272329/37272330/37272331/37272332/37272333/37272334/37272335/37272336/37272337/37272338/37272339/37272340/37272341/37272342/37272343/37272344/37272345/37272346/37272347/37272348/37272349/37272350/37272351/37272352/37272353/37272354/37272355/37272356/37272357/37272358/37272359/37272360/37272361/37272362/37272363/37272364/37272365/37272366/37272367/37272368/37272369/37272370/37272371/37272372/37272373/37272374/37272375/37272376/37272377/37272378/37272379/37272380/37272381/37272382/37272383/37272384/37272385/37272386/37272387/37272388/37272389/37272390/37272391/37272392/37272393/37272394/37272395/37272396/37272397/37272398/37272399/37272400/37272401/37272402/37272403/37272404/37272405/37272406/37272407/37272408/37272409/37272410/37272411/37272412/37272413/37272414/37272415/37272416/37272417/37272418/37272419/37272420/37272421/37272422/37272423/37272424/37272425/37272426/37272427/37272428/37272429/37272430/37272431/37272432/37272433/37272434/37272435/37272436/37272437/37272438/37272439/37272440/37272441/37272442/37272443/37272444/37272445/37272446/37272447/37272448/37272449/37272450/37272451/37272452/37272453/37272454/37272455/37272456/37272457/37272458/37272459/37272460/37272461/37272462/37272463/37272464/37272465/37272466/37272467/37272468/37272469/37272470/37272471/37272472/37272473/37272474/37272475/37272476/37272477/37272478/37272479/37272480/37272481/37272482/37272483/37272484/37272485/37272486/37272487/37272488/37272489/37272490/37272491/37272492/37272493/37272494/37272495/37272496/37272497/37272498/37272499/37272500/37272501/37272502/37272503/37272504/37272505/37272506/37272507/37272508/37272509/37272510/37272511/37272512/37272513/37272514/37272515/37272516/37272517/37272518/37272519/37272520/37272521/37272522/37272523/37272524/37272525/37272526/37272527/37272528/37272529/37272530/37272531/37272532/37272533/37272534/37272535/37272536/37272537/37272538/37272539/37272540/37272541/37272542/37272543/37272544/37272545/37272546/37272547/37272548/37272549/37272550/37272551/37272552/37272553/37272554/37272555/37272556/37272557/37272558/37272559/37272560/37272561/37272562/37272563/37272564/37272565/37272566/37272567/37272568/37272569/37272570/37272571/37272572/37272573/37272574/37272575/37272576/37272577/37272578/37272579/37272580/37272581/37272582/37272583/37272584/37272585/37272586/37272587/37272588/37272589/37272590/37272591/37272592/37272593/37272594/37272595/37272596/37272597/37272598/37272599/37272600/37272601/37272602/37272603/37272604/37272605/37272606/37272607/37272608/37272609/37272610/37272611/37272612/37272613/37272614/37272615/37272616/37272617/37272618/37272619/37272620/37272621/37272622/37272623/37272624/37272625/37272626/37272627/37272628/37272629/37272630/37272631/37272632/37272633/37272634/37272635/37272636/37272637/37272638/37272639/37272640/37272641/37272642/37272643/37272644/37272645/37272646/37272647/37272648/37272649/37272650/37272651/37272652/37272653/37272654/37272655/37272656/37272657/37272658/37272659/37272660/37272661/37272662/37272663/37272664/37272665/37272666/37272667/37272668/37272669/37272670/37272671/37272672/37272673/37272674/37272675/37272676/37272677/37272678/37272679/37272680/37272681/37272682/37272683/37272684/37272685/37272686/37272687/37272688/37272689/37272690/37272691/37272692/37272693/37272694/37272695/37272696/37272697/37272698/37272699/37272700/37272701/37272702/37272703/37272704/37272705/37272706/37272707/37272708/37272709/37272710/37272711/37272712/37272713/37272714/37272715/37272716/37272717/37272718/37272719/37272720/37272721/37272722/37272723/37272724/37272725/37272726/37272727/37272728/37272729/37272730/37272731/37272732/37272733/37272734/37272735/37272736/37272737/37272738/37272739/37272740/37272741/37272742/37272743/37272744/37272745/37272746/37272747/37272748/37272749/37272750/37272751/37272752/37272753/37272754/37272755/37272756/37272757/37272758/37272759/37272760/37272761/37272762/37272763/37272764/37272765/37272766/37272767/37272768/37272769/37272770/37272771/37272772/37272773/37272774/37272775/37272776/37272777/37272778/37272779/37272780/37272781/37272782/37272783/37272784/37272785/37272786/37272787/37272788/37272789/37272790/37272791/37272792/37272793/37272794/37272795/37272796/37272797/37272798/37272799/37272800/37272801/37272802/37272803/37272804/37272805/37272806/37272807/37272808/37272809/37272810/37272811/37272812/37272813/37272814/37272815/37272816/37272817/37272818/37272819/37272820/37272821/37272822/37272823/37272824/37272825/37272826/37272827/37272828/37272829/37272830/37272831/37272832/37272833/37272834/37272835/37272836/37272837/37272838/37272839/37272840/37272841/37272842/37272843/37272844/37272845/37272846/37272847/37272848/37272849/37272850/37272851/37272852/37272853/37272854/37272855/37272856/37272857/37272858/37272859/37272860/37272861/37272862/37272863/37272864/37272865/37272866/37272867/37272868/37272869/37272870/37272871/37272872/37272873/37272874/37272875/37272876/37272877/37272878/37272879/37272880/37272881/37272882/37272883/37272884/37272885/37272886/37272887/37272888/37272889/37272890/37272891/37272892/37272893/37272894/37272895/37272896/37272897/37272898/37272899/37272900/37272901/37272902/37272903/37272904/37272905/37272906/37272907/37272908/37272909/37272910/37272911/37272912/37272913/37272914/37272915/37272916/37272917/37272918/37272919/37272920/37272921/37272922/37272923/37272924/37272925/37272926/37272927/37272928/37272929/37272930/37272931/37272932/37272933/37272934/37272935/37272936/37272937/37272938/37272939/37272940/37272941/37272942/37272943/37272944/37272945/37272946/37272947/37272948/37272949/37272950/37272951/37272952/37272953/37272954/37272955/37272956/37272957/37272958/37272959/37272960/37272961/37272962/37272963/37272964/37272965/37272966/37272967/37272968/37272969/37272970/37272971/37272972/37272973/37272974/37272975/37272976/37272977/37272978/37272979/37272980/37272981/37272982/37272983/37272984/37272985/37272986/37272987/37272988/37272989/37272990/37272991/37272992/37272993/37272994/37272995/37272996/37272997/37272998/37272999/37273000/37273001/37273002/37273003/37273004/37273005/37273006/37273007/37273008/37273009/37273010/37273011/37273012/37273013/37273014/37273015/37273016/37273017/37273018/37273019/37273020/37273021/37273022/37273023/37273024/37273025/37273026/37273027/37273028/37273029/37273030/37273031/37273032/37273033/37273034/37273035/37273036/37273037/37273038/37273039/37273040/37273041/37273042/37273043/37273044/37273045/37273046/37273047/37273048/37273049/37273050/37273051/37273052/37273053/37273054/37273055/37273056/37273057/37273058/37273059/37273060/37273061/37273062/37273063/37273064/37273065/37273066/37273067/37273068/37273069/37273070/37273071/37273072/37273073/37273074/37273075/37273076/37273077/37273078/37273079/37273080/37273081/37273082/37273083/37273084/37273085/37273086/37273087/37273088/37273089/37273090/37273091/37273092/37273093/37273094/37273095/37273096/37273097/37273098/37273099/37273100/37273101/37273102/37273103/37273104/37273105/37273106/37273107/37273108/37273109/37273110/37273111/37273112/37273113/37273114/37273115/37273116/37273117/37273118/37273119/37273120/37273121/37273122/37273123/37273124/37273125/37273126/37273127/37273128/37273129/37273130/37273131/37273132/37273133/37273134/37273135/37273136/37273137/37273138/37273139/37273140/37273141/37273142/37273143/37273144/37273145/37273146/37273147/37273148/37273149/37273150/37273151/37273152/37273153/37273154/37273155/37273156/37273157/37273158/37273159/37273160/37273161/37273162/37273163/37273164/37273165/37273166/37273167/37273168/37273169/37273170/37273171/37273172/37273173/37273174/37273175/37273176/37273177/37273178/37273179/37273180/37273181/37273182/37273183/37273184/37273185/37273186/37273187/37273188/37273189/37273190/37273191/37273192/37273193/37273194/37273195/37273196/37273197/37273198/37273199/37273200/37273201/37273202/37273203/37273204/37273205/37273206/37273207/37273208/37273209/37273210/37273211/37273212/37273213/37273214/37273215/37273216/37273217/37273218/37273219/37273220/37273221/37273222/37273223/37273224/37273225/37273226/37273227/37273228/37273229/37273230/37273231/37273232/37273233/37273234/37273235/37273236/37273237/37273238/37273239/37273240/37273241/37273242/37273243/37273244/37273245/37273246/37273247/37273248/37273249/37273250/37273251/37273252/37273253/37273254/37273255/37273256/37273257/37273258/37273259/37273260/37273261/37273262/37273263/37273264/37273265/37273266/37273267/37273268/37273269/37273270/37273271/37273272/37273273/37273274/37273275/37273276/37273277/37273278/37273279/37273280/37273281/37273282/37273283/37273284/37273285/37273286/37273287/37273288/37273289/37273290/37273291/37273292/37273293/37273294/37273295/37273296/37273297/37273298/37273299/37273300/37273301/37273302/37273303/37273304/37273305/37273306/37273307/37273308/37273309/37273310/37273311/37273312/37273313/37273314/37273315/37273316/37273317/37273318/37273319/37273320/37273321/37273322/37273323/37273324/37273325/37273326/37273327/37273328/37273329/37273330/37273331/37273332/37273333/37273334/37273335/37273336/37273337/37273338/37273339/37273340/37273341/37273342/37273343/37273344/37273345/37273346/37273347/37273348/37273349/37273350/37273351/37273352/37273353/37273354/37273355/37273356/37273357/37273358/37273359/37273360/37273361/37273362/37273363/37273364/37273365/37273366/37273367/37273368/37273369/37273370/37273371/37273372/37273373/37273374/37273375/37273376/37273377/37273378/37273379/37273380/37273381/37273382/37273383/37273384/37273385/37273386/37273387/37273388/37273389/37273390/37273391/37273392/37273393/37273394/37273395/37273396/37273397/37273398/37273399/37273400/37273401/37273402/37273403/37273404/37273405/37273406/37273407/37273408/37273409/37273410/37273411/37273412/37273413/37273414/37273415/37273416/37273417/37273418/37273419/37273420/37273421/37273422/37273423/37273424/37273425/37273426/37273427/37273428/37273429/37273430/37273431/37273432/37273433/37273434/37273435/37273436/37273437/37273438/37273439/37273440/37273441/37273442/37273443/37273444/37273445/37273446/37273447/37273448/37273449/37273450/37273451/37273452/37273453/37273454/37273455/37273456/37273457/37273458/37273459/37273460/37273461/37273462/37273463/37273464/37273465/37273466/37273467/37273468/37273469/37273470/37273471/37273472/37273473/37273474/37273475/37273476/37273477/37273478/37273479/37273480/37273481/37273482/37273483/37273484/37273485/37273486/37273487/37273488/37273489/37273490/37273491/37273492/37273493/37273494/37273495/37273496/37273497/37273498/37273499/37273500/37273501/37273502/37273503/37273504/37273505/37273506/37273507/37273508/37273509/37273510/37273511/37273512/37273513/37273514/37273515/37273516/37273517/37273518/37273519/37273520/37273521/37273522/37273523/37273524/37273525/37273526/37273527/37273528/37273529/37273530/37273531/37273532/37273533/37273534/37273535/37273536/37273537/37273538/37273539/37273540/37273541/37273542/37273543/37273544/37273545/37273546/37273547/37273548/37273549/37273550/37273551/37273552/37273553/37273554/37273555/37273556/37273557/37273558/37273559/37273560/37273561/37273562/37273563/37273564/37273565/37273566/37273567/37273568/37273569/37273570/37273571/37273572/37273573/37273574/37273575/37273576/37273577/37273578/37273579/37273580/37273581/37273582/37273583/37273584/37273585/37273586/37273587/37273588/37273589/37273590/37273591/37273592/37273593/37273594/37273595/37273596/37273597/37273598/37273599/37273600/37273601/37273602/37273603/37273604/37273605/37273606/37273607/37273608/37273609/37273610/37273611/37273612/37273613/37273614/37273615/37273616/37273617/37273618/37273619/37273620/37273621/37273622/37273623/37273624/37273625/37273626/37273627/37273628/37273629/37273630/37273631/37273632/37273633/37273634/37273635/37273636/37273637/37273638/37273639/37273640/37273641/37273642/37273643/37273644/37273645/37273646/37273647/37273648/37273649/37273650/37273651/37273652/37273653/37273654/37273655/37273656/37273657/37273658/37273659/37273660/37273661/37273662/37273663/37273664/37273665/37273666/37273667/37273668/37273669/37273670/37273671/37273672/37273673/37273674/37273675/37273676/37273677/37273678/37273679/37273680/37273681/37273682/37273683/37273684/37273685/37273686/37273687/37273688/37273689/37273690/37273691/37273692/37273693/37273694/37273695/37273696/37273697/37273698/37273699/37273700/37273701/37273702/37273703/37273704/37273705/37273706/37273707/37273708/37273709/37273710/37273711/37273712/37273713/37273714/37273715/37273716/37273717/37273718/37273719/37273720/37273721/37273722/37273723/37273724/37273725/37273726/3727 done 2014-04-01 15:40:10,039.039 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 15:40:10,041.041 INFO:teuthology.task.ceph:mount /dev/sdc on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 15:40:10,041.041 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdc /var/lib/ceph/osd/ceph-5' 2014-04-01 15:40:10,551.551 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:40:11,391.391 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:11.306881 7f7118978780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:11,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:11.784234 7f7118978780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:11,888.888 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:11.802481 7f7118978780 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:40:12,374.374 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:12.287777 7f7118978780 -1 created object store /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal for osd.3 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:12,375.375 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:12.287822 7f7118978780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2014-04-01 15:40:12,375.375 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:12.287896 7f7118978780 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2014-04-01 15:40:12,376.376 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:40:13,167.167 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.082333 7f6652e03780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:13,593.593 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.509668 7f6652e03780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:13,603.603 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.519682 7f6652e03780 -1 filestore(/var/lib/ceph/osd/ceph-4) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:40:14,065.065 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.980061 7f6652e03780 -1 created object store /var/lib/ceph/osd/ceph-4 journal /var/lib/ceph/osd/ceph-4/journal for osd.4 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:14,065.065 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.980108 7f6652e03780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2014-04-01 15:40:14,066.066 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:13.980183 7f6652e03780 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2014-04-01 15:40:14,068.068 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 15:40:14,886.886 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:14.801850 7ffa378ca780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:15,288.288 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:15.204205 7ffa378ca780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:15,306.306 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:15.221317 7ffa378ca780 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 15:40:15,767.767 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:15.682545 7ffa378ca780 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:15,770.770 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:15.682593 7ffa378ca780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2014-04-01 15:40:15,770.770 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 06:40:15.682669 7ffa378ca780 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2014-04-01 15:40:15,770.770 INFO:teuthology.task.ceph:Reading keys from all nodes... 2014-04-01 15:40:15,771.771 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-0/keyring' 2014-04-01 15:40:16,100.100 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-1/keyring' 2014-04-01 15:40:16,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-2/keyring' 2014-04-01 15:40:17,185.185 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/mds/ceph-a/keyring' 2014-04-01 15:40:17,614.614 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-3/keyring' 2014-04-01 15:40:18,125.125 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-4/keyring' 2014-04-01 15:40:18,741.741 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-5/keyring' 2014-04-01 15:40:19,354.354 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /etc/ceph/ceph.client.0.keyring' 2014-04-01 15:40:19,968.968 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'cat -- /etc/ceph/ceph.client.1.keyring' 2014-04-01 15:40:20,581.581 INFO:teuthology.task.ceph:Adding keys to all mons... 2014-04-01 15:40:20,581.581 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo tee -a /etc/ceph/ceph.keyring' 2014-04-01 15:40:21,150.150 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo tee -a /etc/ceph/ceph.keyring' 2014-04-01 15:40:21,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:22,539.539 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:22,897.897 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:23,218.218 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:23,570.570 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:23,935.935 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:24,291.291 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:24,620.620 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:24,979.979 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:25,304.304 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:25,652.652 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:25,977.977 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:26,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:26,701.701 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 15:40:27,051.051 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 15:40:27,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 15:40:27,768.768 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 15:40:28,119.119 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 15:40:28,472.472 INFO:teuthology.task.ceph:Running mkfs on mon nodes... 2014-04-01 15:40:28,472.472 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-a' 2014-04-01 15:40:28,972.972 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i a --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 15:40:29,486.486 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: set fsid to a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:29,658.658 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2014-04-01 15:40:29,661.661 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-c' 2014-04-01 15:40:30,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i c --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 15:40:30,645.645 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: set fsid to a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:30,795.795 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-c for mon.c 2014-04-01 15:40:30,796.796 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-b' 2014-04-01 15:40:31,127.127 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i b --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 15:40:31,642.642 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-mon: set fsid to a7571e41-653a-4300-8fe0-63e1a9732593 2014-04-01 15:40:31,788.788 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-b for mon.b 2014-04-01 15:40:31,789.789 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'rm -- /home/ubuntu/cephtest/monmap /home/ubuntu/cephtest/osdmap' 2014-04-01 15:40:32,282.282 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'rm -- /home/ubuntu/cephtest/monmap /home/ubuntu/cephtest/osdmap' 2014-04-01 15:40:32,608.608 INFO:teuthology.task.ceph:Starting mon daemons... 2014-04-01 15:40:32,608.608 INFO:teuthology.task.ceph.mon.a:Restarting 2014-04-01 15:40:32,609.609 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i a' 2014-04-01 15:40:33,107.107 INFO:teuthology.task.ceph.mon.a:Started 2014-04-01 15:40:33,108.108 INFO:teuthology.task.ceph.mon.c:Restarting 2014-04-01 15:40:33,108.108 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i c' 2014-04-01 15:40:33,430.430 INFO:teuthology.task.ceph.mon.c:Started 2014-04-01 15:40:33,430.430 INFO:teuthology.task.ceph.mon.b:Restarting 2014-04-01 15:40:33,431.431 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i b' 2014-04-01 15:40:33,787.787 INFO:teuthology.task.ceph.mon.b:Started 2014-04-01 15:40:33,788.788 INFO:teuthology.task.ceph:Starting osd daemons... 2014-04-01 15:40:33,788.788 INFO:teuthology.task.ceph.osd.0:Restarting 2014-04-01 15:40:33,789.789 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 0' 2014-04-01 15:40:34,110.110 INFO:teuthology.task.ceph.osd.0:Started 2014-04-01 15:40:34,111.111 INFO:teuthology.task.ceph.osd.1:Restarting 2014-04-01 15:40:34,111.111 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' 2014-04-01 15:40:34,162.162 INFO:teuthology.task.ceph.osd.0.out:[10.214.133.37]: starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2014-04-01 15:40:34,436.436 INFO:teuthology.task.ceph.osd.1:Started 2014-04-01 15:40:34,436.436 INFO:teuthology.task.ceph.osd.2:Restarting 2014-04-01 15:40:34,437.437 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-04-01 15:40:34,483.483 INFO:teuthology.task.ceph.osd.0.err:[10.214.133.37]: 2014-04-01 06:40:34.398545 7f34e540e780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:34,488.488 INFO:teuthology.task.ceph.osd.1.out:[10.214.133.37]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2014-04-01 15:40:34,758.758 INFO:teuthology.task.ceph.osd.2:Started 2014-04-01 15:40:34,758.758 INFO:teuthology.task.ceph.osd.3:Restarting 2014-04-01 15:40:34,759.759 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-04-01 15:40:34,766.766 INFO:teuthology.task.ceph.osd.1.err:[10.214.133.37]: 2014-04-01 06:40:34.681209 7f8fe1820780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:34,810.810 INFO:teuthology.task.ceph.osd.2.out:[10.214.133.37]: starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-04-01 15:40:35,111.111 INFO:teuthology.task.ceph.osd.2.err:[10.214.133.37]: 2014-04-01 06:40:35.021193 7f1caabe0780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:35,217.217 INFO:teuthology.task.ceph.osd.3:Started 2014-04-01 15:40:35,217.217 INFO:teuthology.task.ceph.osd.4:Restarting 2014-04-01 15:40:35,218.218 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4' 2014-04-01 15:40:35,268.268 INFO:teuthology.task.ceph.osd.3.out:[10.214.133.38]: starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-04-01 15:40:35,541.541 INFO:teuthology.task.ceph.osd.4:Started 2014-04-01 15:40:35,542.542 INFO:teuthology.task.ceph.osd.5:Restarting 2014-04-01 15:40:35,542.542 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 5' 2014-04-01 15:40:35,566.566 INFO:teuthology.task.ceph.osd.3.err:[10.214.133.38]: 2014-04-01 06:40:35.481685 7f7a3991c780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:35,592.592 INFO:teuthology.task.ceph.osd.4.out:[10.214.133.38]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal 2014-04-01 15:40:35,863.863 INFO:teuthology.task.ceph.osd.5:Started 2014-04-01 15:40:35,863.863 INFO:teuthology.task.ceph:Starting mds daemons... 2014-04-01 15:40:35,864.864 INFO:teuthology.task.ceph.mds.a:Restarting 2014-04-01 15:40:35,864.864 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f -i a' 2014-04-01 15:40:35,914.914 INFO:teuthology.task.ceph.osd.5.out:[10.214.133.38]: starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal 2014-04-01 15:40:35,916.916 INFO:teuthology.task.ceph.osd.4.err:[10.214.133.38]: 2014-04-01 06:40:35.832279 7fd0410e5780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:36,190.190 INFO:teuthology.task.ceph.mds.a:Started 2014-04-01 15:40:36,191.191 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph mds set_max_mds 1' 2014-04-01 15:40:36,253.253 INFO:teuthology.task.ceph.osd.5.err:[10.214.133.38]: 2014-04-01 06:40:36.168410 7f7d58085780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:40:36,257.257 INFO:teuthology.task.ceph.mds.a.out:[10.214.133.38]: starting mds.a at :/0 2014-04-01 15:40:46,892.892 INFO:teuthology.orchestra.run.err:[10.214.133.37]: max_mds = 1 2014-04-01 15:40:46,903.903 INFO:teuthology.task.ceph:Waiting until ceph is healthy... 2014-04-01 15:40:46,903.903 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 15:40:47,763.763 DEBUG:teuthology.misc:6 of 6 OSDs are up 2014-04-01 15:40:53,761.761 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2014-04-01 15:40:54,384.384 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 72 pgs peering; 72 pgs stuck inactive; 72 pgs stuck unclean 2014-04-01 15:41:01,384.384 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2014-04-01 15:41:01,935.935 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2014-04-01 15:41:01,935.935 INFO:teuthology.run_tasks:Running task thrashosds... 2014-04-01 15:41:01,938.938 INFO:teuthology.task.thrashosds:Beginning thrashosds... 2014-04-01 15:41:01,938.938 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 15:41:02,626.626 INFO:teuthology.task.thrashosds.ceph_manager:[{u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 45, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 0, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}, {u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 0, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 1, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}, {u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 0, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 2, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}] 2014-04-01 15:41:02,628.628 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get data pg_num' 2014-04-01 15:41:03,377.377 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get metadata pg_num' 2014-04-01 15:41:04,066.066 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get rbd pg_num' 2014-04-01 15:41:04,815.815 INFO:teuthology.task.thrashosds.ceph_manager:waiting for clean 2014-04-01 15:41:04,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:05,651.651 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:05,963.963 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:06,681.681 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:06,990.990 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:41:07,704.704 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': []}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 72, u'data_bytes': 1884, u'bytes_used': 1072304128, u'op_per_sec': 3, u'version': 14, u'pgs_by_state': [{u'count': 72, u'state_name': u'active+clean'}], u'write_bytes_sec': 1348, u'bytes_avail': 2801543131136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 5, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:41:07,704.704 INFO:teuthology.task.thrashosds.ceph_manager:clean! 2014-04-01 15:41:07,704.704 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump' 2014-04-01 15:41:08,389.389 INFO:teuthology.task.thrashosds.ceph_manager:['osd.0 up in weight 1 up_from 3 up_thru 4 down_at 0 last_clean_interval [0,0) 10.214.133.37:6800/4936 10.214.133.37:6801/4936 10.214.133.37:6802/4936 10.214.133.37:6803/4936 exists,up c734daeb-bf79-4fdc-900c-73117abb49c0', 'osd.1 up in weight 1 up_from 2 up_thru 4 down_at 0 last_clean_interval [0,0) 10.214.133.37:6805/4943 10.214.133.37:6806/4943 10.214.133.37:6807/4943 10.214.133.37:6808/4943 exists,up 90fd5834-2f7c-4950-a689-12c97fa67457', 'osd.2 up in weight 1 up_from 3 up_thru 3 down_at 0 last_clean_interval [0,0) 10.214.133.37:6810/4986 10.214.133.37:6811/4986 10.214.133.37:6812/4986 10.214.133.37:6813/4986 exists,up 33ed58f5-1178-48a8-be18-55283c1c2e7a', 'osd.3 up in weight 1 up_from 3 up_thru 4 down_at 0 last_clean_interval [0,0) 10.214.133.38:6800/4938 10.214.133.38:6801/4938 10.214.133.38:6802/4938 10.214.133.38:6803/4938 exists,up 6dab52fc-be6f-4ca3-957c-394b904a046b', 'osd.4 up in weight 1 up_from 3 up_thru 3 down_at 0 last_clean_interval [0,0) 10.214.133.38:6805/4947 10.214.133.38:6806/4947 10.214.133.38:6807/4947 10.214.133.38:6808/4947 exists,up 932f00b3-50ea-43d6-aea2-71ea531a3c6e', 'osd.5 up in weight 1 up_from 3 up_thru 3 down_at 0 last_clean_interval [0,0) 10.214.133.38:6810/4980 10.214.133.38:6811/4980 10.214.133.38:6812/4980 10.214.133.38:6813/4980 exists,up 53792d08-9748-4da5-9601-7344a0efe97a'] 2014-04-01 15:41:08,390.390 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -- tell 'mon.*' injectargs '--mon-osd-down-out-interval 0'" 2014-04-01 15:41:09,120.120 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.a: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 15:41:09,229.229 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.b: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 15:41:09,333.333 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.c: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 15:41:09,343.343 INFO:teuthology.run_tasks:Running task radosbench... 2014-04-01 15:41:09,344.344 INFO:teuthology.task.radosbench:Beginning radosbench... 2014-04-01 15:41:09,344.344 INFO:teuthology.task.thrashosds.ceph_manager:creating pool_name unique_pool_0 2014-04-01 15:41:09,345.345 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd erasure-code-profile set teuthologyprofile ruleset-failure-domain=osd m=1 k=2' 2014-04-01 15:41:09,346.346 INFO:teuthology.task.thrashosds.thrasher:starting do_thrash 2014-04-01 15:41:09,346.346 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 15:41:09,346.346 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:09,347.347 INFO:teuthology.task.thrashosds.thrasher:Killing osd 3, live_osds are [1, 0, 3, 2, 5, 4] 2014-04-01 15:41:09,347.347 DEBUG:teuthology.task.ceph.osd.3:waiting for process to exit 2014-04-01 15:41:09,509.509 INFO:teuthology.task.ceph.osd.3:Stopped 2014-04-01 15:41:10,357.357 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool create unique_pool_0 16 16 erasure teuthologyprofile' 2014-04-01 15:41:12,806.806 INFO:teuthology.orchestra.run.err:[10.214.133.37]: pool 'unique_pool_0' created 2014-04-01 15:41:12,818.818 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --name client.0 -p unique_pool_0 bench 1800 write'" 2014-04-01 15:41:13,276.276 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2014-04-01 15:41:13,276.276 INFO:teuthology.task.radosbench:joining radosbench (timing out after 9000s) 2014-04-01 15:41:13,342.342 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Maintaining 16 concurrent writes of 4194304 bytes for up to 1800 seconds or 0 objects 2014-04-01 15:41:13,343.343 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Object prefix: benchmark_data_plana79_5903 2014-04-01 15:41:13,389.389 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:41:13,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 0 0 0 0 0 0 - 0 2014-04-01 15:41:13,391.391 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:41:13.303542 7f82a82f0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/4938 pipe(0x1af31f0 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x1af3450).fault 2014-04-01 15:41:14,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1 16 16 0 0 0 - 0 2014-04-01 15:41:14,507.507 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:14,508.508 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:14,509.509 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 15:41:14,509.509 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:15,329.329 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:15,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2 16 25 9 17.9957 18 0.156574 1.30721 2014-04-01 15:41:16,388.388 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 3 16 26 10 13.3304 4 0.128265 1.18931 2014-04-01 15:41:17,388.388 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 4 16 26 10 9.99792 0 - 1.18931 2014-04-01 15:41:18,388.388 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 5 16 26 10 7.9984 0 - 1.18931 2014-04-01 15:41:19,389.389 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 6 16 26 10 6.66537 0 - 1.18931 2014-04-01 15:41:20,391.391 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 7 16 26 10 5.7132 0 - 1.18931 2014-04-01 15:41:20,790.790 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:20,791.791 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:20,792.792 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 15:41:20,792.792 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:41:20,793.793 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:41:20,793.793 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 15:41:21,389.389 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 8 16 26 10 4.99907 0 - 1.18931 2014-04-01 15:41:22,389.389 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 9 16 26 10 4.44362 0 - 1.18931 2014-04-01 15:41:23,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 10 16 26 10 3.99927 0 - 1.18931 2014-04-01 15:41:24,389.389 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 11 16 26 10 3.6357 0 - 1.18931 2014-04-01 15:41:25,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 12 16 26 10 3.33274 0 - 1.18931 2014-04-01 15:41:26,350.350 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:26,350.350 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:26,351.351 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 15:41:26,351.351 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:26,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 13 16 26 10 3.07638 0 - 1.18931 2014-04-01 15:41:27,053.053 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:27,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 14 16 26 10 2.85664 0 - 1.18931 2014-04-01 15:41:28,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 15 16 26 10 2.6662 0 - 1.18931 2014-04-01 15:41:29,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 16 16 26 10 2.49956 0 - 1.18931 2014-04-01 15:41:30,390.390 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 17 16 26 10 2.35253 0 - 1.18931 2014-04-01 15:41:31,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 18 16 29 13 2.8884 0.8 0.129108 3.54129 2014-04-01 15:41:32,391.391 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:32,392.392 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:32,393.393 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 15:41:32,393.393 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:41:32,393.393 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:41:32,394.394 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 15:41:32,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 19 16 29 13 2.73638 0 - 3.54129 2014-04-01 15:41:33,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:41:33.306182min lat: 0.128265 max lat: 19.9839 avg lat: 7.78262 2014-04-01 15:41:33,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:41:33,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 20 16 37 21 4.1993 16 19.9839 7.78262 2014-04-01 15:41:34,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 21 16 57 41 7.80822 80 0.212425 5.52071 2014-04-01 15:41:35,400.400 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 22 16 65 49 8.90762 32 0.112695 4.66479 2014-04-01 15:41:36,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 23 16 74 58 10.0853 36 0.155013 3.96353 2014-04-01 15:41:37,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 24 16 75 59 9.8317 4 0.118034 3.89835 2014-04-01 15:41:37,962.962 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:37,962.962 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:37,963.963 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 15:41:37,963.963 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:41:37,964.964 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:41:37,964.964 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 15:41:38,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 25 16 75 59 9.43843 0 - 3.89835 2014-04-01 15:41:39,392.392 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 26 16 75 59 9.07542 0 - 3.89835 2014-04-01 15:41:40,392.392 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 27 16 75 59 8.73929 0 - 3.89835 2014-04-01 15:41:41,392.392 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 28 16 75 59 8.42717 0 - 3.89835 2014-04-01 15:41:42,392.392 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 29 16 75 59 8.13658 0 - 3.89835 2014-04-01 15:41:43,391.391 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:43,392.392 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:43,393.393 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 15:41:43,393.393 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:43,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 30 16 75 59 7.86536 0 - 3.89835 2014-04-01 15:41:44,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:44,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 31 16 75 59 7.61164 0 - 3.89835 2014-04-01 15:41:45,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 32 16 75 59 7.37378 0 - 3.89835 2014-04-01 15:41:46,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 33 16 75 59 7.15033 0 - 3.89835 2014-04-01 15:41:47,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 34 16 75 59 6.94002 0 - 3.89835 2014-04-01 15:41:48,393.393 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 35 16 75 59 6.74174 0 - 3.89835 2014-04-01 15:41:49,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 36 16 75 59 6.55447 0 - 3.89835 2014-04-01 15:41:49,615.615 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:49,615.615 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:49,616.616 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 15:41:49,617.617 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:41:50,379.379 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:41:50,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 37 16 75 59 6.37733 0 - 3.89835 2014-04-01 15:41:51,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 38 16 75 59 6.2095 0 - 3.89835 2014-04-01 15:41:52,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 39 16 75 59 6.05028 0 - 3.89835 2014-04-01 15:41:53,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:41:53.309449min lat: 0.084519 max lat: 19.9839 avg lat: 3.89835 2014-04-01 15:41:53,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:41:53,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 40 16 75 59 5.89903 0 - 3.89835 2014-04-01 15:41:54,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 41 16 75 59 5.75514 0 - 3.89835 2014-04-01 15:41:55,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 42 16 75 59 5.61812 0 - 3.89835 2014-04-01 15:41:55,841.841 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:41:55,842.842 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:41:55,842.842 INFO:teuthology.task.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.741098 2014-04-01 15:41:55,843.843 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 3 0.741097842971' 2014-04-01 15:41:56,394.394 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 43 16 75 59 5.48747 0 - 3.89835 2014-04-01 15:41:56,706.706 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.3 primary-affinity to 0.741098 (8485682) 2014-04-01 15:41:57,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 44 16 89 73 6.63529 2.8 0.221593 7.16343 2014-04-01 15:41:58,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 45 16 110 94 8.35421 84 0.718302 5.67386 2014-04-01 15:41:58,707.707 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:41:58.619393 7f82981e0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6810/4980 pipe(0x1af54b0 sd=11 :46826 s=2 pgs=8 cs=1 l=1 c=0x1af5710).injecting socket failure 2014-04-01 15:41:59,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 46 16 134 118 10.2592 96 0.639889 4.59295 2014-04-01 15:41:59,678.678 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:41:59.590621 7f82984e3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x1aef4c0 sd=8 :32827 s=2 pgs=5 cs=1 l=1 c=0x1aef720).injecting socket failure 2014-04-01 15:42:00,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 47 16 157 141 11.9981 92 0.643021 3.90712 2014-04-01 15:42:01,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 48 16 184 168 13.9978 108 0.675025 4.52156 2014-04-01 15:42:01,716.716 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2, 5, 4] 2014-04-01 15:42:01,717.717 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 3 2014-04-01 15:42:01,717.717 INFO:teuthology.task.ceph.osd.3:Restarting 2014-04-01 15:42:01,718.718 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-04-01 15:42:02,178.178 INFO:teuthology.task.ceph.osd.3:Started 2014-04-01 15:42:02,178.178 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-04-01 15:42:02,237.237 INFO:teuthology.task.ceph.osd.3.out:[10.214.133.38]: starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-04-01 15:42:02,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 49 16 202 186 15.1813 72 0.15675 4.14099 2014-04-01 15:42:02,570.570 INFO:teuthology.task.ceph.osd.3.err:[10.214.133.38]: 2014-04-01 06:42:02.485667 7f5d1701a780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:42:02,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 15:42:02,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 15:42:02,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 15:42:02,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 15:42:02,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 15:42:02,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 15:42:02,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 15:42:02,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 15:42:02,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 15:42:02,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 15:42:02,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 15:42:02,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 15:42:02,603.603 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 3, ['dump_ops_in_flight'] 2014-04-01 15:42:03,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 50 16 202 186 14.8777 0 - 4.14099 2014-04-01 15:42:04,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 51 16 202 186 14.586 0 - 4.14099 2014-04-01 15:42:05,395.395 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 52 16 202 186 14.3055 0 - 4.14099 2014-04-01 15:42:06,464.464 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 53 16 207 191 14.4128 5 0.370783 4.07813 2014-04-01 15:42:07,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 54 16 216 200 14.8125 36 0.229993 3.90647 2014-04-01 15:42:07,602.602 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-04-01 15:42:08,165.165 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:42:08,165.165 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:08,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 55 16 220 204 14.834 16 0.068406 3.83145 2014-04-01 15:42:08,927.927 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:09,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 56 16 225 209 14.9262 20 0.23015 3.75653 2014-04-01 15:42:09,539.539 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:10,358.358 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:10,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 57 16 239 223 15.6467 56 0.204371 3.53024 2014-04-01 15:42:11,135.135 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:11,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 58 16 244 228 15.7217 20 0.156474 3.45927 2014-04-01 15:42:11,899.899 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:41:33.169827', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349916, u'health': u'HEALTH_OK', u'kb_used': 8979068, u'store_stats': {u'bytes_total': 1050522, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.870558', u'name': u'b', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439291760, u'health': u'HEALTH_OK', u'kb_used': 9037224, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.487556', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349604, u'health': u'HEALTH_OK', u'kb_used': 8979380, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 226/697 objects degraded (32.425%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 4134756, u'degraded_objects': 226, u'num_pgs': 88, u'recovering_keys_per_sec': 0, u'data_bytes': 918554460, u'degraded_total': 697, u'bytes_used': 2217340928, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 64502207, u'op_per_sec': 15, u'version': 35, u'pgs_by_state': [{u'count': 4, u'state_name': u'active'}, {u'count': 79, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'32.425', u'bytes_avail': 2800398094336}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:11,901.901 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:12,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 59 16 264 248 16.811 80 11.63 3.41337 2014-04-01 15:42:12,591.591 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:13,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:42:13.312056min lat: 0.063861 max lat: 47.1782 avg lat: 3.45576 2014-04-01 15:42:13,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:42:13,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 60 16 275 259 17.264 44 0.748534 3.45576 2014-04-01 15:42:14,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 61 16 287 271 17.7678 48 0.131123 3.4472 2014-04-01 15:42:15,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 62 16 288 272 17.5457 4 13.8782 3.48555 2014-04-01 15:42:15,912.912 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:16,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 63 16 294 278 17.6481 24 0.424698 3.46249 2014-04-01 15:42:16,622.622 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:17,134.134 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:17,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 64 16 294 278 17.3723 0 - 3.46249 2014-04-01 15:42:18,010.010 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:41:33.169827', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349916, u'health': u'HEALTH_OK', u'kb_used': 8979068, u'store_stats': {u'bytes_total': 1050522, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.870558', u'name': u'b', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439291760, u'health': u'HEALTH_OK', u'kb_used': 9037224, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.487556', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349604, u'health': u'HEALTH_OK', u'kb_used': 8979380, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 281/853 objects degraded (32.943%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 281, u'num_pgs': 88, u'data_bytes': 1136658268, u'degraded_total': 853, u'bytes_used': 2409308160, u'op_per_sec': 10, u'write_bytes_sec': 42647694, u'version': 37, u'pgs_by_state': [{u'count': 79, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'32.943', u'bytes_avail': 2800206127104}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:18,011.011 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:18,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 65 16 328 312 19.1971 68 0.451008 3.32695 2014-04-01 15:42:18,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:19,400.400 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 66 16 342 326 19.7546 56 0.711015 3.20822 2014-04-01 15:42:20,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 67 16 362 346 20.6536 80 0.09712 3.06963 2014-04-01 15:42:21,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 68 16 366 350 20.5851 16 0.214404 3.03889 2014-04-01 15:42:22,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:22,397.397 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 69 16 366 350 20.2868 0 - 3.03889 2014-04-01 15:42:22,823.823 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:23,148.148 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:23,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 70 16 366 350 19.997 0 - 3.03889 2014-04-01 15:42:23,927.927 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:41:33.169827', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349916, u'health': u'HEALTH_OK', u'kb_used': 8979068, u'store_stats': {u'bytes_total': 1050522, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.870558', u'name': u'b', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439291760, u'health': u'HEALTH_OK', u'kb_used': 9037224, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.487556', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349604, u'health': u'HEALTH_OK', u'kb_used': 8979380, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 327/1045 objects degraded (31.292%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 327, u'num_pgs': 88, u'data_bytes': 1405093724, u'degraded_total': 1045, u'bytes_used': 2891382784, u'op_per_sec': 8, u'write_bytes_sec': 37741263, u'version': 40, u'pgs_by_state': [{u'count': 79, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'31.292', u'bytes_avail': 2799724052480}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:23,929.929 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:24,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 71 16 366 350 19.7153 0 - 3.03889 2014-04-01 15:42:24,920.920 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:25,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 72 16 366 350 19.4415 0 - 3.03889 2014-04-01 15:42:26,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 73 16 380 364 19.9422 11.2 6.73567 3.13877 2014-04-01 15:42:27,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 74 16 417 401 21.6724 148 0.602799 2.93682 2014-04-01 15:42:28,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:28,398.398 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 75 16 422 406 21.6501 20 0.218095 2.90671 2014-04-01 15:42:29,140.140 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:29,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 76 16 422 406 21.3652 0 - 2.90671 2014-04-01 15:42:29,613.613 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:30,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 77 16 422 406 21.0877 0 - 2.90671 2014-04-01 15:42:30,429.429 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:41:33.169827', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349916, u'health': u'HEALTH_OK', u'kb_used': 8979068, u'store_stats': {u'bytes_total': 1050522, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.870558', u'name': u'b', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439291760, u'health': u'HEALTH_OK', u'kb_used': 9037224, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:41:33.487556', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439349604, u'health': u'HEALTH_OK', u'kb_used': 8979380, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 404/1291 objects degraded (31.294%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 8150109, u'degraded_objects': 404, u'num_pgs': 88, u'recovering_keys_per_sec': 0, u'data_bytes': 1749026652, u'degraded_total': 1291, u'bytes_used': 3278553088, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 52160702, u'op_per_sec': 12, u'version': 43, u'pgs_by_state': [{u'count': 79, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'31.294', u'bytes_avail': 2799336882176}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:30,430.430 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:31,179.179 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:31,401.401 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 78 16 422 406 20.8174 0 - 2.90671 2014-04-01 15:42:32,399.399 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 79 16 424 408 20.6551 2 5.19764 2.91794 2014-04-01 15:42:33,402.402 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:42:33.314939min lat: 0.063861 max lat: 47.1782 avg lat: 2.92177 2014-04-01 15:42:33,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:42:33,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 80 16 451 435 21.7467 108 0.732174 2.92177 2014-04-01 15:42:34,400.400 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 81 16 465 449 22.1695 56 0.101533 2.84496 2014-04-01 15:42:34,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:35,260.260 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:35,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 82 16 465 449 21.8991 0 - 2.84496 2014-04-01 15:42:35,733.733 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:36,401.401 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 83 16 465 449 21.6353 0 - 2.84496 2014-04-01 15:42:36,515.515 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:42:33.170106', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315848, u'health': u'HEALTH_OK', u'kb_used': 9013136, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.487759', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315460, u'health': u'HEALTH_OK', u'kb_used': 9013524, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 357/1366 objects degraded (26.135%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 357, u'num_pgs': 88, u'data_bytes': 1853884252, u'degraded_total': 1366, u'bytes_used': 3616985088, u'op_per_sec': 5, u'write_bytes_sec': 21558528, u'version': 45, u'pgs_by_state': [{u'count': 81, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'26.135', u'bytes_avail': 2798998450176}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:36,516.516 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:37,337.337 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:37,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 84 16 465 449 21.3777 0 - 2.84496 2014-04-01 15:42:38,400.400 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 85 16 465 449 21.1262 0 - 2.84496 2014-04-01 15:42:39,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 86 16 484 468 21.7641 15.2 0.290201 2.93149 2014-04-01 15:42:40,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 87 16 504 488 22.4334 80 1.17825 2.83496 2014-04-01 15:42:40,945.945 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:41,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 88 16 538 522 23.7237 136 0.341521 2.69066 2014-04-01 15:42:41,677.677 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:42,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 89 16 558 542 24.3559 80 0.436331 2.61547 2014-04-01 15:42:42,624.624 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:43,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 90 16 566 550 24.4407 32 0.171096 2.58554 2014-04-01 15:42:43,655.655 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:42:33.170106', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315848, u'health': u'HEALTH_OK', u'kb_used': 9013136, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.487759', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315460, u'health': u'HEALTH_OK', u'kb_used': 9013524, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 323/1423 objects degraded (22.699%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 323, u'num_pgs': 88, u'data_bytes': 1933576028, u'degraded_total': 1423, u'bytes_used': 3812786176, u'op_per_sec': 3, u'write_bytes_sec': 15977439, u'version': 47, u'pgs_by_state': [{u'count': 82, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.699', u'bytes_avail': 2798802649088}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:43,656.656 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:44,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 91 16 566 550 24.1722 0 - 2.58554 2014-04-01 15:42:44,786.786 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:45,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 92 16 566 550 23.9094 0 - 2.58554 2014-04-01 15:42:46,402.402 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 93 16 566 550 23.6523 0 - 2.58554 2014-04-01 15:42:47,401.401 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 94 16 593 577 24.5495 27 0.530693 2.59378 2014-04-01 15:42:48,402.402 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 95 16 621 605 25.4698 112 0.505127 2.50457 2014-04-01 15:42:48,679.679 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:49,223.223 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:49,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 96 16 644 628 26.1627 92 0.761007 2.43384 2014-04-01 15:42:49,858.858 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:50,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 97 16 658 642 26.4702 56 0.169949 2.39553 2014-04-01 15:42:50,652.652 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:42:33.170106', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315848, u'health': u'HEALTH_OK', u'kb_used': 9013136, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.487759', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315460, u'health': u'HEALTH_OK', u'kb_used': 9013524, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 220/1873 objects degraded (11.746%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 220, u'num_pgs': 88, u'data_bytes': 2562721628, u'degraded_total': 1873, u'bytes_used': 4783923200, u'op_per_sec': 9, u'write_bytes_sec': 41567362, u'version': 51, u'pgs_by_state': [{u'count': 85, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.746', u'bytes_avail': 2797831512064}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:50,653.653 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:51,345.345 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:51,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 98 16 660 644 26.2817 8 0.071386 2.38844 2014-04-01 15:42:52,402.402 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 99 16 660 644 26.0163 0 - 2.38844 2014-04-01 15:42:53,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:42:53.318024min lat: 0.063861 max lat: 47.1782 avg lat: 2.38844 2014-04-01 15:42:53,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:42:53,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 100 16 660 644 25.7561 0 - 2.38844 2014-04-01 15:42:54,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 101 16 670 654 25.8971 13.3333 4.97661 2.43006 2014-04-01 15:42:54,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:55,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 102 16 693 677 26.545 92 0.90868 2.39858 2014-04-01 15:42:55,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:56,147.147 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:42:56,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 103 16 716 700 27.1803 92 0.509469 2.34141 2014-04-01 15:42:56,890.890 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:42:33.170106', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315848, u'health': u'HEALTH_OK', u'kb_used': 9013136, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.487759', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315460, u'health': u'HEALTH_OK', u'kb_used': 9013524, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 84/2011 objects degraded (4.177%)'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 84, u'num_pgs': 88, u'data_bytes': 2755659612, u'degraded_total': 2011, u'bytes_used': 5124685824, u'op_per_sec': 9, u'write_bytes_sec': 38900630, u'version': 53, u'pgs_by_state': [{u'count': 87, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.177', u'bytes_avail': 2797490749440}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:42:56,891.891 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:42:57,403.403 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 104 16 743 727 27.9573 108 0.723647 2.28016 2014-04-01 15:42:57,588.588 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:42:58,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 105 16 767 751 28.6052 96 0.697684 2.22716 2014-04-01 15:42:59,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 106 16 794 778 29.3541 108 0.626919 2.17147 2014-04-01 15:43:00,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 107 16 822 806 30.1263 112 0.732907 2.11809 2014-04-01 15:43:01,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:01,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 108 16 848 832 30.8102 104 0.550478 2.07119 2014-04-01 15:43:01,697.697 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:02,181.181 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:43:02,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 109 16 874 858 31.4815 104 0.764688 2.02514 2014-04-01 15:43:03,088.088 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:42:33.170106', u'name': u'a', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315848, u'health': u'HEALTH_OK', u'kb_used': 9013136, u'store_stats': {u'bytes_total': 2099098, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.487759', u'name': u'c', u'avail_percent': 93, u'kb_total': 472345880, u'kb_avail': 439315460, u'health': u'HEALTH_OK', u'kb_used': 9013524, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 88, u'data_bytes': 3200255836, u'bytes_used': 5627695104, u'op_per_sec': 21, u'version': 55, u'pgs_by_state': [{u'count': 88, u'state_name': u'active+clean'}], u'write_bytes_sec': 88190264, u'bytes_avail': 2796987740160}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 14, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:43:03,089.089 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:43:03,089.089 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:03,089.089 INFO:teuthology.task.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5] 2014-04-01 15:43:03,090.090 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 4' 2014-04-01 15:43:03,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 110 16 904 888 32.2861 120 0.411595 1.97619 2014-04-01 15:43:04,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 111 16 928 912 32.8599 96 0.642786 1.93654 2014-04-01 15:43:04,665.665 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.4. 2014-04-01 15:43:05,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 112 16 929 913 32.6022 4 0.381563 1.93484 2014-04-01 15:43:06,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 113 16 931 915 32.3845 8 0.134818 1.93229 2014-04-01 15:43:07,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 114 16 938 922 32.346 28 0.204851 1.93101 2014-04-01 15:43:08,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 115 16 938 922 32.0647 0 - 1.93101 2014-04-01 15:43:09,404.404 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 116 16 938 922 31.7883 0 - 1.93101 2014-04-01 15:43:09,674.674 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5] out_osds: [4] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:09,675.675 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:09,675.675 INFO:teuthology.task.thrashosds.thrasher:Adding osd 4 2014-04-01 15:43:09,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 4' 2014-04-01 15:43:10,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 117 16 938 922 31.5166 0 - 1.93101 2014-04-01 15:43:11,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 118 16 941 925 31.3512 3 7.4489 1.9424 2014-04-01 15:43:11,648.648 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.4. 2014-04-01 15:43:11,658.658 INFO:teuthology.task.thrashosds.thrasher:Added osd 4 2014-04-01 15:43:12,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 119 16 945 929 31.2222 16 5.10736 1.94021 2014-04-01 15:43:13,405.405 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:43:13.320920min lat: 0.063861 max lat: 47.1782 avg lat: 1.94766 2014-04-01 15:43:13,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:43:13,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 120 16 946 930 30.9953 4 8.86254 1.94766 2014-04-01 15:43:14,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 121 16 948 932 30.8053 8 10.7835 1.95574 2014-04-01 15:43:15,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 122 16 968 952 31.2084 80 1.01185 2.02473 2014-04-01 15:43:16,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 123 16 993 977 31.7676 100 0.707002 1.98997 2014-04-01 15:43:16,658.658 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:16,658.658 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:16,659.659 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2014-04-01 15:43:16,660.660 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 1' 2014-04-01 15:43:17,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 124 16 1019 1003 32.35 104 10.0828 1.96139 2014-04-01 15:43:18,210.210 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 1 (8655362) 2014-04-01 15:43:18,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 125 16 1028 1012 32.3791 36 0.439791 1.94902 2014-04-01 15:43:19,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 126 16 1028 1012 32.1221 0 - 1.94902 2014-04-01 15:43:20,406.406 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 127 16 1028 1012 31.8692 0 - 1.94902 2014-04-01 15:43:21,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 128 16 1028 1012 31.6202 0 - 1.94902 2014-04-01 15:43:22,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 129 16 1028 1012 31.3751 0 - 1.94902 2014-04-01 15:43:23,221.221 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:23,221.221 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:23,222.222 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 15:43:23,223.223 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:23,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 130 16 1060 1044 32.1182 25.6 0.607506 1.97185 2014-04-01 15:43:23,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:24,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 131 16 1090 1074 32.789 120 0.468299 1.93048 2014-04-01 15:43:24,457.457 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 24' 2014-04-01 15:43:25,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 132 16 1107 1091 33.0556 68 0.557745 1.92703 2014-04-01 15:43:25,529.529 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 24 2014-04-01 15:43:26,407.407 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 133 16 1113 1097 32.9875 24 0.511346 1.92097 2014-04-01 15:43:27,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 134 16 1113 1097 32.7413 0 - 1.92097 2014-04-01 15:43:28,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 135 16 1113 1097 32.4988 0 - 1.92097 2014-04-01 15:43:29,408.408 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 136 16 1113 1097 32.2598 0 - 1.92097 2014-04-01 15:43:30,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 137 16 1142 1126 32.8709 29 0.473573 1.94353 2014-04-01 15:43:30,535.535 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:30,535.535 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:43:30,536.536 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:31,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:31,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 138 16 1157 1141 33.0675 60 0.437663 1.92805 2014-04-01 15:43:31,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:32,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 139 16 1185 1169 33.6352 112 0.687359 1.8981 2014-04-01 15:43:32,653.653 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:33,016.016 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:43:33,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:43:33.323912min lat: 0.063861 max lat: 47.1782 avg lat: 1.87217 2014-04-01 15:43:33,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:43:33,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 140 16 1211 1195 34.1377 104 0.738485 1.87217 2014-04-01 15:43:33,806.806 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:43:33.170354', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439244432, u'health': u'HEALTH_OK', u'kb_used': 9084552, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:42:33.870805', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439231664, u'health': u'HEALTH_OK', u'kb_used': 9097320, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:43:33.487998', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439243720, u'health': u'HEALTH_OK', u'kb_used': 9085264, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 88, u'data_bytes': 4890560348, u'bytes_used': 8218492928, u'op_per_sec': 12, u'version': 74, u'pgs_by_state': [{u'count': 88, u'state_name': u'active+clean'}], u'write_bytes_sec': 52142691, u'bytes_avail': 2794396942336}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 20, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:43:33,807.807 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:43:33,808.808 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:33,808.808 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 15:43:33,809.809 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:43:33,809.809 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:43:33,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 15:43:34,425.425 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 141 16 1237 1221 34.6331 104 0.446223 1.84291 2014-04-01 15:43:35,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 142 16 1250 1234 34.7553 52 0.355309 1.82977 2014-04-01 15:43:36,409.409 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 143 16 1250 1234 34.5123 0 - 1.82977 2014-04-01 15:43:37,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 144 16 1250 1234 34.2726 0 - 1.82977 2014-04-01 15:43:38,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 145 16 1250 1234 34.0362 0 - 1.82977 2014-04-01 15:43:39,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 146 16 1250 1234 33.8031 0 - 1.82977 2014-04-01 15:43:39,452.452 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:39,453.453 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:39,453.453 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 15:43:39,453.453 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:43:39,453.453 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:43:39,453.453 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 15:43:40,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 147 16 1277 1261 34.3077 21.6 0.534737 1.85941 2014-04-01 15:43:41,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 148 16 1312 1296 35.0218 140 0.25583 1.82411 2014-04-01 15:43:42,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 149 16 1337 1321 35.4577 100 0.434459 1.80198 2014-04-01 15:43:43,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 150 16 1350 1334 35.568 52 0.967392 1.7927 2014-04-01 15:43:44,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 151 16 1364 1348 35.7032 56 0.197408 1.78173 2014-04-01 15:43:45,261.261 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:45,261.261 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:43:45,262.262 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:45,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 152 16 1366 1350 35.521 8 0.533448 1.77993 2014-04-01 15:43:46,062.062 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:46,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 153 16 1366 1350 35.2888 0 - 1.77993 2014-04-01 15:43:46,631.631 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:47,420.420 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 154 16 1366 1350 35.0596 0 - 1.77993 2014-04-01 15:43:47,472.472 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:48,178.178 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:43:48,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 155 16 1366 1350 34.8335 0 - 1.77993 2014-04-01 15:43:49,087.087 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:43:33.170354', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439244432, u'health': u'HEALTH_OK', u'kb_used': 9084552, u'store_stats': {u'bytes_total': 3147674, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:43:33.871040', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439131488, u'health': u'HEALTH_OK', u'kb_used': 9197496, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:43:33.487998', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439243720, u'health': u'HEALTH_OK', u'kb_used': 9085264, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 88, u'data_bytes': 5700061020, u'bytes_used': 9572720640, u'op_per_sec': 10, u'version': 80, u'pgs_by_state': [{u'count': 88, u'state_name': u'active+clean'}], u'write_bytes_sec': 42111951, u'bytes_avail': 2793042714624}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 20, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:43:49,088.088 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:43:49,088.088 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:49,088.088 INFO:teuthology.task.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 5, 4] 2014-04-01 15:43:49,089.089 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3' 2014-04-01 15:43:49,419.419 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 156 16 1366 1350 34.6102 0 - 1.77993 2014-04-01 15:43:50,376.376 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.3. 2014-04-01 15:43:50,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 157 16 1391 1375 35.0266 20 0.42604 1.82536 2014-04-01 15:43:51,411.411 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 158 16 1404 1388 35.1339 52 0.069361 1.8099 2014-04-01 15:43:52,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 159 16 1410 1394 35.0639 24 1.64553 1.80811 2014-04-01 15:43:53,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:43:53.326881min lat: 0.063861 max lat: 47.1782 avg lat: 1.80878 2014-04-01 15:43:53,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:43:53,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 160 16 1427 1411 35.2697 68 0.504059 1.80878 2014-04-01 15:43:54,419.419 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 161 16 1458 1442 35.8207 124 0.50962 1.78187 2014-04-01 15:43:55,385.385 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4] out_osds: [3] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:43:55,386.386 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:43:55,387.387 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 15:43:55,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:43:55,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 162 16 1461 1445 35.6736 12 1.01301 1.78019 2014-04-01 15:43:56,272.272 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:43:56,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 163 16 1461 1445 35.4548 0 - 1.78019 2014-04-01 15:43:56,924.924 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 24' 2014-04-01 15:43:57,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 164 16 1461 1445 35.2386 0 - 1.78019 2014-04-01 15:43:58,226.226 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 24 2014-04-01 15:43:58,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 165 16 1461 1445 35.025 0 - 1.78019 2014-04-01 15:43:59,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 166 16 1470 1454 35.0309 9 5.22522 1.80165 2014-04-01 15:44:00,412.412 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 167 16 1478 1462 35.0127 32 0.713583 1.80461 2014-04-01 15:44:01,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 168 16 1496 1480 35.2328 72 1.43046 1.81219 2014-04-01 15:44:02,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 169 16 1520 1504 35.5923 96 0.704197 1.79429 2014-04-01 15:44:03,234.234 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4] out_osds: [3] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:44:03,234.234 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:44:03,235.235 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 15:44:03,236.236 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:03,413.413 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 170 16 1526 1510 35.5241 24 0.745356 1.78952 2014-04-01 15:44:03,842.842 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:04,413.413 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 171 16 1526 1510 35.3163 0 - 1.78952 2014-04-01 15:44:04,472.472 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 24' 2014-04-01 15:44:05,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 172 16 1526 1510 35.111 0 - 1.78952 2014-04-01 15:44:05,504.504 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 24 2014-04-01 15:44:06,413.413 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 173 16 1526 1510 34.9081 0 - 1.78952 2014-04-01 15:44:07,413.413 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 174 16 1526 1510 34.7074 0 - 1.78952 2014-04-01 15:44:08,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 175 16 1551 1535 35.0805 20 0.896463 1.82021 2014-04-01 15:44:09,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 176 16 1580 1564 35.5401 116 0.454255 1.79856 2014-04-01 15:44:10,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 177 16 1591 1575 35.5879 44 1.18013 1.79089 2014-04-01 15:44:10,517.517 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4] out_osds: [3] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:44:10,517.517 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:44:10,518.518 INFO:teuthology.task.thrashosds.thrasher:Adding osd 3 2014-04-01 15:44:10,519.519 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 3' 2014-04-01 15:44:11,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 178 16 1594 1578 35.4554 12 1.9059 1.79055 2014-04-01 15:44:12,035.035 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.3. 2014-04-01 15:44:12,046.046 INFO:teuthology.task.thrashosds.thrasher:Added osd 3 2014-04-01 15:44:12,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 179 16 1594 1578 35.2573 0 - 1.79055 2014-04-01 15:44:13,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:44:13.329718min lat: 0.063861 max lat: 47.1782 avg lat: 1.79055 2014-04-01 15:44:13,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:44:13,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 180 16 1594 1578 35.0614 0 - 1.79055 2014-04-01 15:44:14,414.414 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 181 16 1594 1578 34.8677 0 - 1.79055 2014-04-01 15:44:15,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 182 16 1598 1582 34.764 4 4.71615 1.7991 2014-04-01 15:44:16,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 183 16 1611 1595 34.8582 52 0.22411 1.81237 2014-04-01 15:44:17,044.044 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 2, 5, 4, 3] 2014-04-01 15:44:17,045.045 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:44:17,045.045 INFO:teuthology.task.thrashosds.thrasher:Killing osd 2, live_osds are [1, 0, 2, 5, 4, 3] 2014-04-01 15:44:17,046.046 DEBUG:teuthology.task.ceph.osd.2:waiting for process to exit 2014-04-01 15:44:17,214.214 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:44:17.123951 7f82992e8700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6810/4986 pipe(0x7f82a0018350 sd=14 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a00185b0).fault 2014-04-01 15:44:17,215.215 INFO:teuthology.task.ceph.osd.2:Stopped 2014-04-01 15:44:17,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 184 16 1613 1597 34.7122 8 1.88629 1.81269 2014-04-01 15:44:18,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 185 16 1625 1609 34.784 48 0.571798 1.81227 2014-04-01 15:44:19,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 186 16 1637 1621 34.855 48 0.121148 1.81051 2014-04-01 15:44:20,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 187 16 1641 1625 34.7542 16 0.344866 1.80673 2014-04-01 15:44:21,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 188 16 1641 1625 34.5693 0 - 1.80673 2014-04-01 15:44:22,213.213 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-04-01 15:44:22,214.214 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:44:22,215.215 INFO:teuthology.task.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.869132 2014-04-01 15:44:22,215.215 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 5 0.86913162221' 2014-04-01 15:44:22,415.415 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 189 16 1641 1625 34.3864 0 - 1.80673 2014-04-01 15:44:23,176.176 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.5 primary-affinity to 0.869132 (8569592) 2014-04-01 15:44:23,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 190 16 1641 1625 34.2054 0 - 1.80673 2014-04-01 15:44:24,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 191 16 1641 1625 34.0263 0 - 1.80673 2014-04-01 15:44:25,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 192 16 1642 1626 33.87 0.8 14.9933 1.81484 2014-04-01 15:44:26,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 193 16 1645 1629 33.7566 12 0.108996 1.8218 2014-04-01 15:44:27,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 194 16 1645 1629 33.5826 0 - 1.8218 2014-04-01 15:44:28,185.185 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-04-01 15:44:28,185.185 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:44:28,186.186 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 15:44:28,186.186 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:28,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 195 16 1646 1630 33.4309 2 17.6497 1.83151 2014-04-01 15:44:29,001.001 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:30,371.371 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 196 16 1646 1630 33.2603 0 - 1.83151 2014-04-01 15:44:30,382.382 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:44:30,382.382 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 34' 2014-04-01 15:44:30,416.416 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 197 16 1646 1630 33.0915 0 - 1.83151 2014-04-01 15:44:31,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 198 16 1646 1630 32.9244 0 - 1.83151 2014-04-01 15:44:31,780.780 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pg_num to 34 2014-04-01 15:44:32,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 199 16 1646 1630 32.7589 0 - 1.83151 2014-04-01 15:44:33,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:44:33.332698min lat: 0.063861 max lat: 47.1782 avg lat: 1.83151 2014-04-01 15:44:33,417.417 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:44:33,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 200 16 1646 1630 32.5951 0 - 1.83151 2014-04-01 15:44:34,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 201 16 1646 1630 32.433 0 - 1.83151 2014-04-01 15:44:35,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 202 16 1646 1630 32.2724 0 - 1.83151 2014-04-01 15:44:36,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 203 16 1646 1630 32.1134 0 - 1.83151 2014-04-01 15:44:36,791.791 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 5, 4, 3] 2014-04-01 15:44:36,792.792 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 2 2014-04-01 15:44:36,792.792 INFO:teuthology.task.ceph.osd.2:Restarting 2014-04-01 15:44:36,793.793 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-04-01 15:44:37,116.116 INFO:teuthology.task.ceph.osd.2:Started 2014-04-01 15:44:37,116.116 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 15:44:37,336.336 INFO:teuthology.task.ceph.osd.2.out:[10.214.133.37]: starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-04-01 15:44:37,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 204 16 1646 1630 31.956 0 - 1.83151 2014-04-01 15:44:37,531.531 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 15:44:37,531.531 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 15:44:37,531.531 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 15:44:37,532.532 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 15:44:37,532.532 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 15:44:37,532.532 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 15:44:37,532.532 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 15:44:37,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 15:44:37,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 15:44:37,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 15:44:37,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 15:44:37,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 15:44:37,538.538 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 2, ['dump_ops_in_flight'] 2014-04-01 15:44:37,850.850 INFO:teuthology.task.ceph.osd.2.err:[10.214.133.37]: 2014-04-01 06:44:37.765565 7f67a1472780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 15:44:38,418.418 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 205 16 1646 1630 31.8001 0 - 1.83151 2014-04-01 15:44:39,419.419 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 206 16 1646 1630 31.6457 0 - 1.83151 2014-04-01 15:44:40,419.419 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 207 16 1646 1630 31.4929 0 - 1.83151 2014-04-01 15:44:41,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 208 16 1651 1635 31.4376 1.53846 23.0986 1.89615 2014-04-01 15:44:42,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 209 16 1679 1663 31.823 112 0.408372 1.99133 2014-04-01 15:44:42,536.536 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 15:44:42,970.970 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:44:42,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:43,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 210 16 1709 1693 32.2428 120 0.28608 1.98204 2014-04-01 15:44:43,903.903 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:44,385.385 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:44,423.423 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 211 16 1729 1713 32.4691 80 0.373149 1.96663 2014-04-01 15:44:45,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 212 16 1749 1733 32.6932 80 0.558246 1.95312 2014-04-01 15:44:46,420.420 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 213 16 1760 1744 32.7463 44 0.16705 1.94345 2014-04-01 15:44:47,419.419 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 214 16 1761 1745 32.6119 4 0.125305 1.94241 2014-04-01 15:44:48,262.262 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:48,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 215 16 1761 1745 32.4603 0 - 1.94241 2014-04-01 15:44:48,683.683 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:44:49,401.401 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:44:33.170633', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188300, u'health': u'HEALTH_OK', u'kb_used': 9140684, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.894013', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439058224, u'health': u'HEALTH_OK', u'kb_used': 9270760, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.488307', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188228, u'health': u'HEALTH_OK', u'kb_used': 9140756, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 99/5215 objects degraded (1.898%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 99, u'num_pgs': 98, u'data_bytes': 7235176284, u'degraded_total': 5215, u'bytes_used': 11916320768, u'op_per_sec': 16, u'write_bytes_sec': 67378943, u'version': 114, u'pgs_by_state': [{u'count': 95, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.898', u'bytes_avail': 2790699114496}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 34, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:44:49,401.401 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:49,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 216 16 1761 1745 32.31 0 - 1.94241 2014-04-01 15:44:50,153.153 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:50,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 217 16 1761 1745 32.1611 0 - 1.94241 2014-04-01 15:44:51,423.423 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 218 16 1791 1775 32.5639 30 0.737064 1.96329 2014-04-01 15:44:52,425.425 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 219 16 1819 1803 32.9266 112 0.546099 1.94228 2014-04-01 15:44:53,420.420 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:44:53.335816min lat: 0.063861 max lat: 47.1782 avg lat: 1.9211 2014-04-01 15:44:53,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:44:53,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 220 16 1845 1829 33.2496 104 0.637066 1.9211 2014-04-01 15:44:53,614.614 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:54,340.340 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:54,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 221 16 1857 1841 33.3163 48 0.612632 1.91296 2014-04-01 15:44:54,825.825 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:44:55,420.420 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 222 16 1857 1841 33.1662 0 - 1.91296 2014-04-01 15:44:55,531.531 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:44:33.170633', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188300, u'health': u'HEALTH_OK', u'kb_used': 9140684, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.894013', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439058224, u'health': u'HEALTH_OK', u'kb_used': 9270760, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.488307', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188228, u'health': u'HEALTH_OK', u'kb_used': 9140756, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 103/5311 objects degraded (1.939%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 103, u'num_pgs': 98, u'data_bytes': 7369394012, u'degraded_total': 5311, u'bytes_used': 12299739136, u'op_per_sec': 4, u'write_bytes_sec': 20129438, u'version': 117, u'pgs_by_state': [{u'count': 95, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.939', u'bytes_avail': 2790315696128}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 34, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:44:55,532.532 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:44:56,306.306 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:44:56,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 223 16 1857 1841 33.0175 0 - 1.91296 2014-04-01 15:44:57,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 224 16 1857 1841 32.8701 0 - 1.91296 2014-04-01 15:44:58,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 225 16 1857 1841 32.724 0 - 1.91296 2014-04-01 15:44:59,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 226 16 1882 1866 33.0216 20 0.622832 1.93482 2014-04-01 15:44:59,641.641 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:00,407.407 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:00,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 227 16 1910 1894 33.3694 112 0.480528 1.91476 2014-04-01 15:45:01,067.067 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:45:01,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 228 16 1931 1915 33.5914 84 0.446237 1.90136 2014-04-01 15:45:01,968.968 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:44:33.170633', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188300, u'health': u'HEALTH_OK', u'kb_used': 9140684, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.894013', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439058224, u'health': u'HEALTH_OK', u'kb_used': 9270760, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.488307', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188228, u'health': u'HEALTH_OK', u'kb_used': 9140756, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 36/5680 objects degraded (0.634%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 36, u'num_pgs': 98, u'data_bytes': 7885293404, u'degraded_total': 5680, u'bytes_used': 12846747648, u'op_per_sec': 6, u'write_bytes_sec': 28508679, u'version': 120, u'pgs_by_state': [{u'count': 97, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.634', u'bytes_avail': 2789768687616}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 34, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:45:01,969.969 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:02,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 229 16 1962 1946 33.9862 124 0.318381 1.88068 2014-04-01 15:45:02,731.731 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:03,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 230 16 1984 1968 34.221 88 0.431473 1.86618 2014-04-01 15:45:04,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 231 16 1986 1970 34.1074 8 0.623514 1.8649 2014-04-01 15:45:05,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 232 16 1986 1970 33.9604 0 - 1.8649 2014-04-01 15:45:06,209.209 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:06,423.423 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 233 16 1986 1970 33.8147 0 - 1.8649 2014-04-01 15:45:06,916.916 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:07,390.390 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:45:07,422.422 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 234 16 1986 1970 33.6702 0 - 1.8649 2014-04-01 15:45:08,159.159 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.036327', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.111323', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:44:33.170633', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188300, u'health': u'HEALTH_OK', u'kb_used': 9140684, u'store_stats': {u'bytes_total': 5244826, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.894013', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439058224, u'health': u'HEALTH_OK', u'kb_used': 9270760, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:44:33.488307', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439188228, u'health': u'HEALTH_OK', u'kb_used': 9140756, u'store_stats': {u'bytes_total': 7341978, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 98, u'data_bytes': 8279557980, u'bytes_used': 13421883392, u'op_per_sec': 17, u'version': 122, u'pgs_by_state': [{u'count': 98, u'state_name': u'active+clean'}], u'write_bytes_sec': 75396051, u'bytes_avail': 2789193551872}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 34, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:45:08,159.159 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:45:08,160.160 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:08,160.160 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 1 2014-04-01 15:45:08,160.160 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:45:08,161.161 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:45:08,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-04-01 15:45:08,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 235 16 2009 1993 33.9183 23 0.337885 1.88431 2014-04-01 15:45:09,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 236 16 2036 2020 34.2322 108 0.615105 1.86776 2014-04-01 15:45:10,423.423 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 237 16 2058 2042 34.459 88 0.703076 1.85305 2014-04-01 15:45:11,423.423 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 238 16 2078 2062 34.6503 80 0.687999 1.84234 2014-04-01 15:45:12,426.426 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 239 16 2098 2082 34.84 80 0.134733 1.83307 2014-04-01 15:45:13,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:45:13.338895min lat: 0.063861 max lat: 47.1782 avg lat: 1.82879 2014-04-01 15:45:13,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:45:13,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 240 16 2104 2088 34.7948 24 0.345951 1.82879 2014-04-01 15:45:13,744.744 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:13,745.745 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:13,745.745 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 15:45:13,746.746 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:14,517.517 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:14,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 241 16 2104 2088 34.6504 0 - 1.82879 2014-04-01 15:45:14,987.987 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:45:14,988.988 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pg_num 34' 2014-04-01 15:45:15,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 242 16 2104 2088 34.5072 0 - 1.82879 2014-04-01 15:45:16,196.196 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pg_num to 34 2014-04-01 15:45:16,425.425 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 243 16 2104 2088 34.3652 0 - 1.82879 2014-04-01 15:45:17,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 244 16 2104 2088 34.2244 0 - 1.82879 2014-04-01 15:45:18,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 245 16 2127 2111 34.4601 18.4 0.251449 1.85542 2014-04-01 15:45:19,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 246 16 2160 2144 34.8565 132 0.242767 1.8344 2014-04-01 15:45:20,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 247 16 2182 2166 35.0716 88 0.455089 1.82286 2014-04-01 15:45:21,207.207 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:21,208.208 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:21,209.209 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 15:45:21,209.209 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:21,426.426 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 248 16 2186 2170 34.9947 16 0.106371 1.81991 2014-04-01 15:45:21,984.984 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:22,425.425 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 249 16 2188 2172 34.8863 8 0.873295 1.81915 2014-04-01 15:45:23,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 250 16 2188 2172 34.7468 0 - 1.81915 2014-04-01 15:45:24,425.425 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 251 16 2188 2172 34.6083 0 - 1.81915 2014-04-01 15:45:25,424.424 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 252 16 2188 2172 34.471 0 - 1.81915 2014-04-01 15:45:26,431.431 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 253 16 2218 2202 34.809 30 0.219434 1.83725 2014-04-01 15:45:27,431.431 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 254 16 2237 2221 34.9711 76 0.978997 1.82884 2014-04-01 15:45:27,446.446 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:27,446.446 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:27,446.446 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 15:45:27,447.447 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:28,230.230 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:28,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 255 16 2258 2242 35.1633 84 0.774184 1.81698 2014-04-01 15:45:28,726.726 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:45:28,727.727 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 26' 2014-04-01 15:45:29,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 256 16 2279 2263 35.354 84 0.227046 1.80576 2014-04-01 15:45:29,764.764 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 26 2014-04-01 15:45:30,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 257 16 2295 2279 35.4655 64 0.306431 1.79834 2014-04-01 15:45:31,426.426 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 258 16 2298 2282 35.3745 12 0.124568 1.79639 2014-04-01 15:45:32,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 259 16 2298 2282 35.2379 0 - 1.79639 2014-04-01 15:45:33,426.426 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:45:33.342082min lat: 0.063861 max lat: 47.1782 avg lat: 1.79639 2014-04-01 15:45:33,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:45:33,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 260 16 2298 2282 35.1024 0 - 1.79639 2014-04-01 15:45:34,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 261 16 2298 2282 34.9679 0 - 1.79639 2014-04-01 15:45:34,773.773 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:34,773.773 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:34,773.773 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 15:45:34,773.773 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:35,570.570 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:35,774.774 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 262 16 2298 2282 34.8344 0 - 1.79639 2014-04-01 15:45:36,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 263 16 2298 2282 34.702 0 - 1.79639 2014-04-01 15:45:37,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 264 16 2298 2282 34.5705 0 - 1.79639 2014-04-01 15:45:38,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 265 16 2298 2282 34.4401 0 - 1.79639 2014-04-01 15:45:39,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 266 16 2298 2282 34.3106 0 - 1.79639 2014-04-01 15:45:40,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 267 16 2298 2282 34.1821 0 - 1.79639 2014-04-01 15:45:41,388.388 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:41,388.388 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:45:41,389.389 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 4 2014-04-01 15:45:41,390.390 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:45:41,390.390 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:45:41,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-04-01 15:45:41,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 268 16 2298 2282 34.0546 0 - 1.79639 2014-04-01 15:45:42,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 269 16 2298 2282 33.928 0 - 1.79639 2014-04-01 15:45:43,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 270 16 2298 2282 33.8023 0 - 1.79639 2014-04-01 15:45:44,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 271 16 2298 2282 33.6776 0 - 1.79639 2014-04-01 15:45:45,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 272 16 2298 2282 33.5538 0 - 1.79639 2014-04-01 15:45:46,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 273 16 2298 2282 33.4309 0 - 1.79639 2014-04-01 15:45:46,929.929 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:45:46,930.930 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:45:46,931.931 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:47,428.428 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 274 16 2298 2282 33.3089 0 - 1.79639 2014-04-01 15:45:47,632.632 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:48,263.263 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:48,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 275 16 2305 2289 33.2895 1.64706 0.113105 1.81441 2014-04-01 15:45:48,998.998 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:49,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 276 16 2305 2289 33.1689 0 - 1.81441 2014-04-01 15:45:49,630.630 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:45:50,395.395 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:45:33.204271', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128776, u'health': u'HEALTH_OK', u'kb_used': 9200208, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.894297', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438976348, u'health': u'HEALTH_OK', u'kb_used': 9352636, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.488589', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128756, u'health': u'HEALTH_OK', u'kb_used': 9200228, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 8550726, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 13719570788, u'bytes_used': 15391727616, u'recovering_objects_per_sec': 2, u'op_per_sec': 61, u'version': 143, u'pgs_by_state': [{u'count': 3, u'state_name': u'creating'}, {u'count': 15, u'state_name': u'peering'}, {u'count': 100, u'state_name': u'active+clean'}], u'write_bytes_sec': 256521790, u'bytes_avail': 2787223707648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 38, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:45:50,396.396 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:50,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 277 16 2305 2289 33.0492 0 - 1.81441 2014-04-01 15:45:51,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 278 16 2305 2289 32.9303 0 - 1.81441 2014-04-01 15:45:52,119.119 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:52,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 279 16 2305 2289 32.8123 0 - 1.81441 2014-04-01 15:45:53,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:45:53.344981min lat: 0.063861 max lat: 47.1782 avg lat: 1.81441 2014-04-01 15:45:53,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:45:53,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 280 16 2305 2289 32.6951 0 - 1.81441 2014-04-01 15:45:54,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 281 16 2305 2289 32.5787 0 - 1.81441 2014-04-01 15:45:55,429.429 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 282 16 2305 2289 32.4632 0 - 1.81441 2014-04-01 15:45:55,740.740 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:56,529.529 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:56,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 283 16 2305 2289 32.3485 0 - 1.81441 2014-04-01 15:45:57,156.156 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:45:57,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 284 16 2305 2289 32.2346 0 - 1.81441 2014-04-01 15:45:57,938.938 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:45:33.204271', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128776, u'health': u'HEALTH_OK', u'kb_used': 9200208, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.894297', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438976348, u'health': u'HEALTH_OK', u'kb_used': 9352636, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.488589', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128756, u'health': u'HEALTH_OK', u'kb_used': 9200228, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 118, u'data_bytes': 14302579044, u'bytes_used': 15560945664, u'op_per_sec': 0, u'version': 146, u'pgs_by_state': [{u'count': 2, u'state_name': u'creating'}, {u'count': 14, u'state_name': u'peering'}, {u'count': 102, u'state_name': u'active+clean'}], u'write_bytes_sec': 714505, u'bytes_avail': 2787054489600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 38, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:45:57,939.939 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:45:58,430.430 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 285 16 2305 2289 32.1215 0 - 1.81441 2014-04-01 15:45:58,639.639 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:45:59,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 286 16 2315 2299 32.149 3.63636 29.669 1.93728 2014-04-01 15:46:00,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 287 16 2339 2323 32.3714 96 0.668954 1.97283 2014-04-01 15:46:01,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 288 16 2368 2352 32.6618 116 0.607615 1.95773 2014-04-01 15:46:02,254.254 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:02,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 289 16 2403 2387 33.0331 140 0.172049 1.9361 2014-04-01 15:46:02,827.827 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:03,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 290 16 2423 2407 33.195 80 0.384502 1.92626 2014-04-01 15:46:03,645.645 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:46:04,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 291 16 2447 2431 33.4108 96 0.611211 1.91335 2014-04-01 15:46:04,579.579 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:45:33.204271', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128776, u'health': u'HEALTH_OK', u'kb_used': 9200208, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.894297', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438976348, u'health': u'HEALTH_OK', u'kb_used': 9352636, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:45:33.488589', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439128756, u'health': u'HEALTH_OK', u'kb_used': 9200228, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'21 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 14163755, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 15665727844, u'bytes_used': 15569522688, u'recovering_objects_per_sec': 3, u'op_per_sec': 135, u'version': 149, u'pgs_by_state': [{u'count': 118, u'state_name': u'active+clean'}], u'write_bytes_sec': 570091150, u'bytes_avail': 2787045912576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 38, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:46:04,580.580 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:46:04,580.580 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:46:04,581.581 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 15:46:04,581.581 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:05,354.354 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:05,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 292 16 2478 2462 33.721 124 0.544982 1.89626 2014-04-01 15:46:06,071.071 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pgp_num 26' 2014-04-01 15:46:06,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 293 16 2501 2485 33.9199 92 0.660769 1.88537 2014-04-01 15:46:07,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 294 16 2531 2515 34.2126 120 0.509746 1.86823 2014-04-01 15:46:07,803.803 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pgp_num to 26 2014-04-01 15:46:08,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 295 16 2546 2530 34.3 60 0.441563 1.86109 2014-04-01 15:46:09,432.432 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 296 16 2550 2534 34.2381 16 1.62179 1.86057 2014-04-01 15:46:10,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 297 16 2555 2539 34.1902 20 0.776811 1.85844 2014-04-01 15:46:11,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 298 16 2561 2545 34.156 24 0.101171 1.85449 2014-04-01 15:46:12,432.432 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 299 16 2561 2545 34.0417 0 - 1.85449 2014-04-01 15:46:12,815.815 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:46:12,816.816 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:46:12,816.816 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 15:46:12,817.817 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:13,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:46:13.347602min lat: 0.063861 max lat: 47.1782 avg lat: 1.86034 2014-04-01 15:46:13,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:46:13,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 300 16 2583 2567 34.2216 44 0.414319 1.86034 2014-04-01 15:46:13,584.584 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:14,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 34' 2014-04-01 15:46:14,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 301 16 2596 2580 34.2806 52 0.422223 1.85535 2014-04-01 15:46:15,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 302 16 2612 2596 34.379 64 0.510323 1.85044 2014-04-01 15:46:15,611.611 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 34 2014-04-01 15:46:16,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 303 16 2615 2599 34.3051 12 0.199403 1.84861 2014-04-01 15:46:17,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 304 16 2630 2614 34.3896 60 0.929448 1.84672 2014-04-01 15:46:18,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 305 16 2634 2618 34.3293 16 1.11714 1.8449 2014-04-01 15:46:19,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 306 16 2638 2622 34.2694 16 0.233089 1.84288 2014-04-01 15:46:20,433.433 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 307 16 2638 2622 34.1578 0 - 1.84288 2014-04-01 15:46:20,626.626 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:46:20,627.627 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:46:20,627.627 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 15:46:20,628.628 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:46:20,628.628 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:46:20,629.629 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 15:46:21,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 308 16 2654 2638 34.2546 32 0.152192 1.84904 2014-04-01 15:46:22,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 309 16 2672 2656 34.3768 72 0.951259 1.84149 2014-04-01 15:46:23,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 310 16 2687 2671 34.4594 60 0.714239 1.83598 2014-04-01 15:46:24,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 311 16 2687 2671 34.3486 0 - 1.83598 2014-04-01 15:46:25,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 312 16 2694 2678 34.3282 14 0.130046 1.83475 2014-04-01 15:46:26,098.098 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:46:26,099.099 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:46:26,100.100 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool data 2014-04-01 15:46:26,100.100 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:26,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 313 16 2698 2682 34.2697 16 0.245556 1.83596 2014-04-01 15:46:26,884.884 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:27,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 314 16 2701 2685 34.1987 12 4.43325 1.83737 2014-04-01 15:46:27,513.513 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 24' 2014-04-01 15:46:28,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 315 16 2714 2698 34.2552 52 1.96566 1.84175 2014-04-01 15:46:28,694.694 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pgp_num to 24 2014-04-01 15:46:29,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 316 16 2717 2701 34.1848 12 1.49426 1.8408 2014-04-01 15:46:30,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 317 16 2734 2718 34.2914 68 0.295489 1.83981 2014-04-01 15:46:31,434.434 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 318 16 2738 2722 34.2339 16 1.00985 1.83824 2014-04-01 15:46:32,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 319 16 2750 2734 34.277 48 0.315958 1.83551 2014-04-01 15:46:33,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:46:33.350285min lat: 0.063861 max lat: 47.1782 avg lat: 1.82992 2014-04-01 15:46:33,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:46:33,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 320 16 2770 2754 34.4199 80 0.600145 1.82992 2014-04-01 15:46:33,705.705 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:46:33,706.706 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:46:33,706.706 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:34,431.431 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:34,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 321 16 2773 2757 34.3501 12 1.64378 1.82905 2014-04-01 15:46:35,063.063 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:35,435.435 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 322 16 2777 2761 34.2931 16 1.73141 1.82878 2014-04-01 15:46:35,847.847 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:36,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 323 16 2777 2761 34.1869 0 - 1.82878 2014-04-01 15:46:36,622.622 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:46:37,392.392 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4669/12546 objects degraded (37.215%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 9519236, u'degraded_objects': 4669, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17465084260, u'degraded_total': 12546, u'bytes_used': 18813108224, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 26653861, u'op_per_sec': 6, u'version': 171, u'pgs_by_state': [{u'count': 107, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'37.215', u'bytes_avail': 2783802327040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:46:37,393.393 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:37,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 324 16 2782 2766 34.1431 10 0.118692 1.82918 2014-04-01 15:46:38,136.136 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:38,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 325 16 2795 2779 34.198 52 0.197472 1.83578 2014-04-01 15:46:39,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 326 16 2799 2783 34.1422 16 0.220562 1.83352 2014-04-01 15:46:40,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 327 16 2806 2790 34.1234 28 2.42457 1.83427 2014-04-01 15:46:41,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 328 16 2813 2797 34.1047 28 2.88149 1.835 2014-04-01 15:46:41,755.755 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:42,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 329 16 2816 2800 34.0375 12 0.136167 1.83456 2014-04-01 15:46:42,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:43,231.231 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:46:43,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 330 16 2820 2804 33.9829 16 0.388459 1.83282 2014-04-01 15:46:44,007.007 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4599/12681 objects degraded (36.267%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 32689680, u'degraded_objects': 4599, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17653827940, u'degraded_total': 12681, u'bytes_used': 19397758976, u'recovering_objects_per_sec': 7, u'write_bytes_sec': 45402334, u'op_per_sec': 10, u'version': 175, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'36.267', u'bytes_avail': 2783217676288}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:46:44,008.008 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:44,436.436 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 331 16 2824 2808 33.9285 16 3.85652 1.83526 2014-04-01 15:46:44,705.705 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:45,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 332 16 2826 2810 33.8504 8 0.245145 1.83548 2014-04-01 15:46:46,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 333 16 2827 2811 33.7608 4 6.5633 1.83716 2014-04-01 15:46:47,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 334 16 2828 2812 33.6717 4 6.20789 1.83871 2014-04-01 15:46:48,313.313 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:48,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 335 16 2828 2812 33.5712 0 - 1.83871 2014-04-01 15:46:48,943.943 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:49,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 336 16 2828 2812 33.4712 0 - 1.83871 2014-04-01 15:46:49,727.727 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:46:50,437.437 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 337 16 2828 2812 33.3719 0 - 1.83871 2014-04-01 15:46:50,448.448 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4533/12741 objects degraded (35.578%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 16147307, u'degraded_objects': 4533, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17737714020, u'degraded_total': 12741, u'bytes_used': 19765190656, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 5382435, u'op_per_sec': 1, u'version': 179, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'35.578', u'bytes_avail': 2782850244608}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:46:50,449.449 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:51,268.268 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:51,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 338 16 2832 2816 33.3205 4 10.0607 1.84956 2014-04-01 15:46:52,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 339 16 2833 2817 33.234 4 0.86016 1.84921 2014-04-01 15:46:53,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:46:53.353001min lat: 0.063861 max lat: 47.1782 avg lat: 1.84921 2014-04-01 15:46:53,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:46:53,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 340 16 2833 2817 33.1363 0 - 1.84921 2014-04-01 15:46:54,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 341 16 2833 2817 33.0391 0 - 1.84921 2014-04-01 15:46:54,898.898 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:55,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 342 16 2833 2817 32.9425 0 - 1.84921 2014-04-01 15:46:55,706.706 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:56,339.339 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:46:56,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 343 16 2833 2817 32.8465 0 - 1.84921 2014-04-01 15:46:57,063.063 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4418/12756 objects degraded (34.635%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 17639765, u'degraded_objects': 4418, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17758685540, u'degraded_total': 12756, u'bytes_used': 20043575296, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 2204970, u'op_per_sec': 0, u'version': 183, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'34.635', u'bytes_avail': 2782571859968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:46:57,065.065 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:46:57,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 344 16 2833 2817 32.751 0 - 1.84921 2014-04-01 15:46:57,760.760 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:46:58,438.438 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 345 16 2833 2817 32.6561 0 - 1.84921 2014-04-01 15:46:59,441.441 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 346 16 2833 2817 32.5617 0 - 1.84921 2014-04-01 15:47:00,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 347 16 2833 2817 32.4678 0 - 1.84921 2014-04-01 15:47:01,367.367 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:01,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 348 16 2833 2817 32.3746 0 - 1.84921 2014-04-01 15:47:02,198.198 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:02,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 349 16 2835 2819 32.3047 0.8 18.2085 1.85843 2014-04-01 15:47:02,839.839 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:03,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 350 16 2837 2821 32.2353 8 0.359147 1.86464 2014-04-01 15:47:03,558.558 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4289/12759 objects degraded (33.615%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 20984128, u'degraded_objects': 4289, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17762879844, u'degraded_total': 12759, u'bytes_used': 20226084864, u'recovering_objects_per_sec': 5, u'version': 186, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'33.615', u'bytes_avail': 2782389350400}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:03,559.559 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:04,271.271 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:04,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 351 16 2840 2824 32.1776 12 1.17629 1.87175 2014-04-01 15:47:05,439.439 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 352 16 2840 2824 32.0862 0 - 1.87175 2014-04-01 15:47:06,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 353 16 2840 2824 31.9953 0 - 1.87175 2014-04-01 15:47:07,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 354 16 2840 2824 31.9049 0 - 1.87175 2014-04-01 15:47:07,897.897 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:08,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 355 16 2840 2824 31.815 0 - 1.87175 2014-04-01 15:47:08,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:09,371.371 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:09,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 356 16 2840 2824 31.7257 0 - 1.87175 2014-04-01 15:47:11,177.177 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 357 16 2840 2824 31.6368 0 - 1.87175 2014-04-01 15:47:11,178.178 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4209/12786 objects degraded (32.919%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 18145442, u'degraded_objects': 4209, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17800628580, u'degraded_total': 12786, u'bytes_used': 20512321536, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 5336894, u'op_per_sec': 1, u'version': 190, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'32.919', u'bytes_avail': 2782103113728}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:11,179.179 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:11,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 358 16 2840 2824 31.5484 0 - 1.87175 2014-04-01 15:47:11,886.886 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:12,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 359 16 2840 2824 31.4606 0 - 1.87175 2014-04-01 15:47:13,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:47:13.355695min lat: 0.063861 max lat: 47.1782 avg lat: 1.87175 2014-04-01 15:47:13,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:47:13,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 360 16 2840 2824 31.3732 0 - 1.87175 2014-04-01 15:47:14,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 361 16 2840 2824 31.2863 0 - 1.87175 2014-04-01 15:47:15,440.440 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 362 16 2840 2824 31.1998 0 - 1.87175 2014-04-01 15:47:15,497.497 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:16,342.342 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:16,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 363 16 2840 2824 31.1139 0 - 1.87175 2014-04-01 15:47:17,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:17,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 364 16 2840 2824 31.0284 0 - 1.87175 2014-04-01 15:47:17,777.777 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4040/12786 objects degraded (31.597%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 30313942, u'degraded_objects': 4040, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17800628580, u'degraded_total': 12786, u'bytes_used': 20846825472, u'recovering_objects_per_sec': 7, u'version': 194, u'pgs_by_state': [{u'count': 108, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'31.597', u'bytes_avail': 2781768609792}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:17,778.778 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:18,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 365 16 2840 2824 30.9434 0 - 1.87175 2014-04-01 15:47:18,496.496 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:19,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 366 16 2840 2824 30.8589 0 - 1.87175 2014-04-01 15:47:21,210.210 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 367 16 2840 2824 30.7748 0 - 1.87175 2014-04-01 15:47:21,441.441 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 368 16 2844 2828 30.7346 0.941176 16.8212 1.90063 2014-04-01 15:47:22,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:22,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 369 16 2845 2829 30.6622 4 38.9888 1.91374 2014-04-01 15:47:22,900.900 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:23,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 370 16 2845 2829 30.5793 0 - 1.91374 2014-04-01 15:47:23,542.542 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:24,313.313 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3798/12795 objects degraded (29.683%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 20726190, u'degraded_objects': 3798, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17813211492, u'degraded_total': 12795, u'bytes_used': 21061652480, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 2590773, u'op_per_sec': 0, u'version': 196, u'pgs_by_state': [{u'count': 109, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'29.683', u'bytes_avail': 2781553782784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:24,314.314 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:24,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 371 16 2845 2829 30.4969 0 - 1.91374 2014-04-01 15:47:25,068.068 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:25,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 372 16 2845 2829 30.4149 0 - 1.91374 2014-04-01 15:47:26,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 373 16 2845 2829 30.3334 0 - 1.91374 2014-04-01 15:47:27,441.441 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 374 16 2850 2834 30.3057 4 0.242665 1.94206 2014-04-01 15:47:28,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 375 16 2852 2836 30.2462 8 36.9688 1.95429 2014-04-01 15:47:28,691.691 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:29,450.450 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:29,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 376 16 2867 2851 30.3253 60 1.05889 2.02228 2014-04-01 15:47:30,071.071 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:30,442.442 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 377 16 2869 2853 30.2661 8 0.769907 2.02149 2014-04-01 15:47:30,795.795 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:46:33.204481', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065600, u'health': u'HEALTH_OK', u'kb_used': 9263384, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.894493', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438913856, u'health': u'HEALTH_OK', u'kb_used': 9415128, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:46:33.488841', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439065248, u'health': u'HEALTH_OK', u'kb_used': 9263736, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3091/12813 objects degraded (24.124%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 11920495, u'degraded_objects': 3091, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17838377316, u'degraded_total': 12813, u'bytes_used': 20363341824, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 5960247, u'op_per_sec': 1, u'version': 199, u'pgs_by_state': [{u'count': 111, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'24.124', u'bytes_avail': 2782252093440}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:30,796.796 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:31,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 378 16 2873 2857 30.2284 16 2.55575 2.02194 2014-04-01 15:47:31,562.562 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:32,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 379 16 2875 2859 30.1697 8 0.14293 2.02177 2014-04-01 15:47:33,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:47:33.358495min lat: 0.063861 max lat: 47.1782 avg lat: 2.02109 2014-04-01 15:47:33,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:47:33,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 380 16 2876 2860 30.1009 4 0.079089 2.02109 2014-04-01 15:47:34,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 381 16 2876 2860 30.0218 0 - 2.02109 2014-04-01 15:47:35,172.172 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:35,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 382 16 2876 2860 29.9433 0 - 2.02109 2014-04-01 15:47:36,038.038 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:36,443.443 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 383 16 2876 2860 29.8651 0 - 2.02109 2014-04-01 15:47:36,817.817 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:37,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 384 16 2876 2860 29.7873 0 - 2.02109 2014-04-01 15:47:37,832.832 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2688/12891 objects degraded (20.852%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 27259294, u'degraded_objects': 2688, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17947429220, u'degraded_total': 12891, u'bytes_used': 20172304384, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 10903717, u'op_per_sec': 2, u'version': 204, u'pgs_by_state': [{u'count': 112, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.852', u'bytes_avail': 2782443130880}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:37,833.833 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:38,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 385 16 2878 2862 29.7307 1.6 9.09974 2.02603 2014-04-01 15:47:38,674.674 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:39,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 386 16 2880 2864 29.6744 8 9.37121 2.02794 2014-04-01 15:47:40,444.444 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 387 16 2880 2864 29.5977 0 - 2.02794 2014-04-01 15:47:41,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 388 16 2882 2866 29.5421 4 0.234377 2.05931 2014-04-01 15:47:42,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 389 16 2882 2866 29.4661 0 - 2.05931 2014-04-01 15:47:42,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:43,272.272 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:43,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 390 16 2886 2870 29.4316 8 0.109629 2.06584 2014-04-01 15:47:43,906.906 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:44,445.445 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 391 16 2892 2876 29.4177 24 14.3724 2.07919 2014-04-01 15:47:44,674.674 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2631/12909 objects degraded (20.381%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 41978868, u'degraded_objects': 2631, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 17972595044, u'degraded_total': 12909, u'bytes_used': 20222980096, u'recovering_objects_per_sec': 10, u'write_bytes_sec': 9125840, u'op_per_sec': 2, u'version': 207, u'pgs_by_state': [{u'count': 112, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.381', u'bytes_avail': 2782392455168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:44,675.675 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:45,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 392 16 2899 2883 29.4141 28 2.23284 2.09028 2014-04-01 15:47:45,448.448 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:46,445.445 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 393 16 2899 2883 29.3392 0 - 2.09028 2014-04-01 15:47:47,445.445 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 394 16 2901 2885 29.2851 4 0.346786 2.12336 2014-04-01 15:47:48,445.445 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 395 16 2912 2896 29.3223 44 0.11223 2.13638 2014-04-01 15:47:49,068.068 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:49,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 396 16 2921 2905 29.3392 36 3.99886 2.13725 2014-04-01 15:47:49,507.507 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:47:49.420951 7f7598496700 0 -- 10.214.133.37:0/1010425 >> 10.214.133.37:6789/0 pipe(0x7f75940193f0 sd=7 :45525 s=1 pgs=0 cs=0 l=1 c=0x7f7594019650).fault 2014-04-01 15:47:49,776.776 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:50,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:50,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 397 16 2923 2907 29.2854 8 2.41987 2.13666 2014-04-01 15:47:51,281.281 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2597/12972 objects degraded (20.020%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 4321174, u'degraded_objects': 2597, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18060675428, u'degraded_total': 12972, u'bytes_used': 20493234176, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 11883230, u'op_per_sec': 2, u'version': 211, u'pgs_by_state': [{u'count': 112, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.020', u'bytes_avail': 2782122201088}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:51,282.282 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:51,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 398 16 2936 2920 29.3425 52 3.39305 2.13989 2014-04-01 15:47:52,003.003 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:52,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 399 16 2951 2935 29.4193 60 4.48514 2.13669 2014-04-01 15:47:53,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:47:53.361144min lat: 0.063861 max lat: 99.2686 avg lat: 2.13581 2014-04-01 15:47:53,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:47:53,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 400 16 2953 2937 29.3657 8 1.41369 2.13581 2014-04-01 15:47:54,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 401 16 2953 2937 29.2925 0 - 2.13581 2014-04-01 15:47:55,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 402 16 2957 2941 29.2594 8 0.605898 2.13645 2014-04-01 15:47:55,626.626 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:56,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 403 16 2957 2941 29.1868 0 - 2.13645 2014-04-01 15:47:56,774.774 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:57,396.396 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:47:57,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 404 16 2957 2941 29.1146 0 - 2.13645 2014-04-01 15:47:58,183.183 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2504/13131 objects degraded (19.069%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 37727753, u'degraded_objects': 2504, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18282973540, u'degraded_total': 13131, u'bytes_used': 21070774272, u'recovering_objects_per_sec': 8, u'write_bytes_sec': 34134634, u'op_per_sec': 8, u'version': 216, u'pgs_by_state': [{u'count': 112, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'19.069', u'bytes_avail': 2781544660992}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:47:58,183.183 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:47:58,446.446 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 405 16 2959 2943 29.0624 2.66667 5.86771 2.13889 2014-04-01 15:47:58,869.869 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:47:59,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 406 16 2962 2946 29.0204 12 7.02046 2.14437 2014-04-01 15:48:00,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 407 16 2963 2947 28.9589 4 0.102429 2.14368 2014-04-01 15:48:01,447.447 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 408 16 2963 2947 28.8879 0 - 2.14368 2014-04-01 15:48:02,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 409 16 2963 2947 28.8173 0 - 2.14368 2014-04-01 15:48:02,497.497 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:03,216.216 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:03,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 410 16 2963 2947 28.747 0 - 2.14368 2014-04-01 15:48:03,843.843 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:04,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 411 16 2970 2954 28.7452 7 0.213619 2.15411 2014-04-01 15:48:04,573.573 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2445/13143 objects degraded (18.603%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 37893586, u'degraded_objects': 2445, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18299750756, u'degraded_total': 13143, u'bytes_used': 21207019520, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 7217826, u'op_per_sec': 1, u'version': 219, u'pgs_by_state': [{u'count': 112, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'18.603', u'bytes_avail': 2781408415744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:04,574.574 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:05,301.301 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:05,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 412 16 2977 2961 28.7434 28 0.191752 2.16325 2014-04-01 15:48:06,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 413 16 2994 2978 28.8384 68 0.445481 2.17543 2014-04-01 15:48:07,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 414 16 3004 2988 28.8654 40 0.392032 2.17151 2014-04-01 15:48:08,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 415 16 3025 3009 28.9982 84 0.22484 2.16472 2014-04-01 15:48:08,910.910 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:09,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 416 16 3033 3017 29.0054 32 0.462742 2.16036 2014-04-01 15:48:09,622.622 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:10,260.260 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:10,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 417 16 3037 3021 28.9742 16 1.65061 2.15977 2014-04-01 15:48:10,991.991 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2071/13248 objects degraded (15.633%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 8614059, u'degraded_objects': 2071, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18446551396, u'degraded_total': 13248, u'bytes_used': 21392510976, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 22611906, u'op_per_sec': 5, u'version': 223, u'pgs_by_state': [{u'count': 113, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.633', u'bytes_avail': 2781222924288}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:10,992.992 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:11,448.448 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 418 16 3037 3021 28.9049 0 - 2.15977 2014-04-01 15:48:11,744.744 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:12,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 419 16 3037 3021 28.8359 0 - 2.15977 2014-04-01 15:48:13,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:48:13.363852min lat: 0.063861 max lat: 99.2686 avg lat: 2.16194 2014-04-01 15:48:13,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:48:13,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 420 16 3041 3025 28.8053 5.33333 3.1255 2.16194 2014-04-01 15:48:14,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 421 16 3047 3031 28.7939 24 5.64397 2.16926 2014-04-01 15:48:15,407.407 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:15,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 422 16 3075 3059 28.9911 112 0.407448 2.16454 2014-04-01 15:48:16,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:16,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 423 16 3086 3070 29.0265 44 1.10408 2.15964 2014-04-01 15:48:16,836.836 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:17,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 424 16 3094 3078 29.0335 32 2.08951 2.15804 2014-04-01 15:48:17,879.879 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2032/13476 objects degraded (15.079%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 19943263, u'degraded_objects': 2032, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18765318500, u'degraded_total': 13476, u'bytes_used': 21620822016, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 54390718, u'op_per_sec': 12, u'version': 228, u'pgs_by_state': [{u'count': 113, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.079', u'bytes_avail': 2780994613248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:17,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:18,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 425 16 3101 3085 29.0311 28 1.87591 2.15557 2014-04-01 15:48:18,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:19,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 426 16 3111 3095 29.0568 40 1.77641 2.1539 2014-04-01 15:48:20,449.449 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 427 16 3120 3104 29.0731 36 0.689874 2.14934 2014-04-01 15:48:21,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 428 16 3120 3104 29.0051 0 - 2.14934 2014-04-01 15:48:22,209.209 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:22,450.450 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 429 16 3120 3104 28.9375 0 - 2.14934 2014-04-01 15:48:22,781.781 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:23,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 430 16 3147 3131 29.1214 36 0.53431 2.19638 2014-04-01 15:48:23,650.650 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:24,445.445 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2004/13620 objects degraded (14.714%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 27518549, u'degraded_objects': 2004, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 18966645092, u'degraded_total': 13620, u'bytes_used': 21961752576, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 60540809, u'op_per_sec': 14, u'version': 231, u'pgs_by_state': [{u'count': 113, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.714', u'bytes_avail': 2780653682688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:24,446.446 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:24,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 431 16 3175 3159 29.3136 112 0.604512 2.18051 2014-04-01 15:48:25,197.197 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:25,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 432 16 3186 3170 29.3476 44 0.935902 2.17657 2014-04-01 15:48:26,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 433 16 3197 3181 29.3814 44 0.203107 2.17106 2014-04-01 15:48:27,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 434 16 3199 3183 29.3322 8 0.523363 2.17002 2014-04-01 15:48:28,450.450 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 435 16 3199 3183 29.2647 0 - 2.17002 2014-04-01 15:48:28,852.852 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:29,561.561 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:29,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 436 16 3199 3183 29.1976 0 - 2.17002 2014-04-01 15:48:30,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:30,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 437 16 3199 3183 29.1308 0 - 2.17002 2014-04-01 15:48:31,275.275 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:47:33.204778', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987544, u'health': u'HEALTH_OK', u'kb_used': 9341440, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.894729', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438873444, u'health': u'HEALTH_OK', u'kb_used': 9455540, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:47:33.489070', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438987504, u'health': u'HEALTH_OK', u'kb_used': 9341480, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1641/13845 objects degraded (11.853%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 5387932, u'degraded_objects': 1641, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 19281217892, u'degraded_total': 13845, u'bytes_used': 22156267520, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 42025874, u'op_per_sec': 10, u'version': 235, u'pgs_by_state': [{u'count': 114, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.853', u'bytes_avail': 2780459167744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:31,276.276 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:31,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 438 16 3217 3201 29.2286 18 5.24133 2.18653 2014-04-01 15:48:32,022.022 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:32,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 439 16 3246 3230 29.4263 116 0.745322 2.17295 2014-04-01 15:48:32,549.549 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:48:32.464466 7f82986e5700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6805/4947 pipe(0x7f82a000ade0 sd=13 :57066 s=2 pgs=8 cs=1 l=1 c=0x7f82a000b040).injecting socket failure 2014-04-01 15:48:33,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:48:33.366668min lat: 0.063861 max lat: 135.172 avg lat: 2.16777 2014-04-01 15:48:33,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:48:33,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 440 16 3258 3242 29.4685 48 0.347633 2.16777 2014-04-01 15:48:34,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 441 16 3267 3251 29.4832 36 0.367738 2.16524 2014-04-01 15:48:35,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 442 16 3272 3256 29.4618 20 0.152866 2.16267 2014-04-01 15:48:35,665.665 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:36,563.563 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:36,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 443 16 3273 3257 29.4043 4 0.577115 2.16218 2014-04-01 15:48:37,189.189 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:37,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 444 16 3273 3257 29.3381 0 - 2.16218 2014-04-01 15:48:37,911.911 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1279/14100 objects degraded (9.071%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 21957896, u'degraded_objects': 1279, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 19637733732, u'degraded_total': 14100, u'bytes_used': 22630641664, u'recovering_objects_per_sec': 5, u'write_bytes_sec': 96980707, u'op_per_sec': 23, u'version': 240, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.071', u'bytes_avail': 2779984793600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:37,912.912 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:38,452.452 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 445 16 3273 3257 29.2722 0 - 2.16218 2014-04-01 15:48:38,662.662 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:39,298.298 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:48:39.213315 7f829a7f3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6804/5446 pipe(0x1b0f980 sd=11 :35462 s=2 pgs=2 cs=1 l=1 c=0x1afbf10).injecting socket failure 2014-04-01 15:48:39,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 446 16 3281 3265 29.2783 10.6667 5.80029 2.17253 2014-04-01 15:48:39,764.764 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:48:39.669987 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a0020750 sd=12 :57401 s=2 pgs=2 cs=1 l=1 c=0x7f82a000dff0).injecting socket failure 2014-04-01 15:48:40,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 447 16 3309 3293 29.4633 112 0.862172 2.17059 2014-04-01 15:48:41,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 448 16 3332 3316 29.6029 92 0.325964 2.16001 2014-04-01 15:48:42,283.283 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:42,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 449 16 3338 3322 29.5904 24 1.9695 2.15783 2014-04-01 15:48:42,850.850 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:43,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 450 16 3355 3339 29.6757 68 1.69552 2.15451 2014-04-01 15:48:43,631.631 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:44,349.349 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1254/14202 objects degraded (8.830%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 26895264, u'degraded_objects': 1254, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 19780340068, u'degraded_total': 14202, u'bytes_used': 22701375488, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 55583547, u'op_per_sec': 13, u'version': 243, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.830', u'bytes_avail': 2779914059776}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:44,350.350 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:44,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 451 16 3355 3339 29.6099 0 - 2.15451 2014-04-01 15:48:45,249.249 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:45,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 452 16 3356 3340 29.5532 2 1.68273 2.15437 2014-04-01 15:48:46,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 453 16 3356 3340 29.488 0 - 2.15437 2014-04-01 15:48:47,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 454 16 3369 3353 29.5376 26 4.14727 2.16004 2014-04-01 15:48:48,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 455 16 3384 3368 29.6045 60 0.272174 2.1585 2014-04-01 15:48:48,873.873 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:49,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 456 16 3411 3395 29.7764 108 0.635712 2.14785 2014-04-01 15:48:49,457.457 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:50,242.242 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:50,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 457 16 3429 3413 29.8688 72 0.884308 2.14078 2014-04-01 15:48:51,042.042 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1217/14379 objects degraded (8.464%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 9747311, u'degraded_objects': 1217, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 20027804004, u'degraded_total': 14379, u'bytes_used': 23084048384, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 31408004, u'op_per_sec': 7, u'version': 247, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.464', u'bytes_avail': 2779531386880}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:51,043.043 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:51,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 458 16 3435 3419 29.856 24 1.25379 2.13889 2014-04-01 15:48:51,815.815 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:52,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 459 16 3435 3419 29.7909 0 - 2.13889 2014-04-01 15:48:53,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:48:53.369310min lat: 0.063861 max lat: 135.172 avg lat: 2.13889 2014-04-01 15:48:53,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:48:53,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 460 16 3435 3419 29.7261 0 - 2.13889 2014-04-01 15:48:54,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 461 16 3435 3419 29.6617 0 - 2.13889 2014-04-01 15:48:55,431.431 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:55,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 462 16 3435 3419 29.5975 0 - 2.13889 2014-04-01 15:48:56,079.079 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:56,459.459 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 463 16 3455 3439 29.7063 16 0.231176 2.1517 2014-04-01 15:48:56,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:48:57,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 464 16 3471 3455 29.7802 64 0.152744 2.14518 2014-04-01 15:48:57,880.880 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1196/14607 objects degraded (8.188%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 9270106, u'degraded_objects': 1196, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 20346571108, u'degraded_total': 14607, u'bytes_used': 23649009664, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 18540212, u'op_per_sec': 4, u'version': 252, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.188', u'bytes_avail': 2778966425600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:48:57,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:48:58,454.454 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 465 16 3490 3474 29.8796 76 0.807372 2.13898 2014-04-01 15:48:58,658.658 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:48:59,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 466 16 3517 3501 30.0472 108 0.573745 2.12864 2014-04-01 15:49:00,459.459 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 467 16 3532 3516 30.1113 60 1.20508 2.12276 2014-04-01 15:49:01,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 468 16 3535 3519 30.0726 12 2.01222 2.12275 2014-04-01 15:49:02,291.291 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:02,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 469 16 3546 3530 30.1023 44 0.9362 2.12261 2014-04-01 15:49:03,009.009 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:03,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 470 16 3546 3530 30.0382 0 - 2.12261 2014-04-01 15:49:03,635.635 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:04,414.414 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1174/14865 objects degraded (7.898%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 36200812, u'degraded_objects': 1174, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 20707281252, u'degraded_total': 14865, u'bytes_used': 24157343744, u'recovering_objects_per_sec': 8, u'write_bytes_sec': 119462682, u'op_per_sec': 28, u'version': 255, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.898', u'bytes_avail': 2778458091520}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:04,415.415 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:04,455.455 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 471 16 3546 3530 29.9744 0 - 2.12261 2014-04-01 15:49:05,220.220 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:05,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 472 16 3552 3536 29.9618 8 0.322699 2.12306 2014-04-01 15:49:06,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 473 16 3563 3547 29.9914 44 3.83523 2.13055 2014-04-01 15:49:07,460.460 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 474 16 3590 3574 30.156 108 0.343957 2.12115 2014-04-01 15:49:08,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 475 16 3608 3592 30.2441 72 0.476813 2.11372 2014-04-01 15:49:08,838.838 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:09,399.399 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:09,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 476 16 3615 3599 30.2393 28 1.35657 2.11147 2014-04-01 15:49:10,321.321 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:10,462.462 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:49:10.369480 7f82a83f1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a0017910 sd=9 :33211 s=2 pgs=8 cs=1 l=1 c=0x7f82a001c2c0).injecting socket failure 2014-04-01 15:49:10,463.463 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 477 16 3619 3603 30.2095 16 1.87914 2.11144 2014-04-01 15:49:11,046.046 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1144/14961 objects degraded (7.647%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 4280722, u'degraded_objects': 1144, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 20841498980, u'degraded_total': 14961, u'bytes_used': 24604512256, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 26754513, u'op_per_sec': 6, u'version': 259, u'pgs_by_state': [{u'count': 115, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.647', u'bytes_avail': 2778010923008}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:11,047.047 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:11,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 478 16 3625 3609 30.1965 24 0.160134 2.10942 2014-04-01 15:49:11,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:12,459.459 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 479 16 3650 3634 30.3422 100 0.150622 2.10807 2014-04-01 15:49:13,459.459 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:49:13.371985min lat: 0.063861 max lat: 135.172 avg lat: 2.10124 2014-04-01 15:49:13,460.460 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:49:13,460.460 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 480 16 3667 3651 30.4206 68 0.331921 2.10124 2014-04-01 15:49:14,456.456 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 481 16 3677 3661 30.4405 40 0.163382 2.09708 2014-04-01 15:49:15,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:15,738.738 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 482 16 3681 3665 30.4106 16 2.13942 2.09756 2014-04-01 15:49:16,565.565 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:17,323.323 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 483 16 3694 3678 30.4552 52 0.845178 2.09708 2014-04-01 15:49:17,936.936 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 484 16 3701 3685 30.4502 28 0.194443 2.09505 2014-04-01 15:49:18,281.281 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:18,457.457 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 485 16 3704 3688 30.4121 12 0.195304 2.09355 2014-04-01 15:49:18,993.993 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 462/15315 objects degraded (3.017%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 462, u'num_pgs': 118, u'data_bytes': 21336426852, u'degraded_total': 15315, u'bytes_used': 25109446656, u'op_per_sec': 18, u'write_bytes_sec': 77270103, u'version': 264, u'pgs_by_state': [{u'count': 117, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.017', u'bytes_avail': 2777505988608}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:18,994.994 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:19,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 486 16 3704 3688 30.3495 0 - 2.09355 2014-04-01 15:49:19,832.832 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:20,482.482 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 487 16 3722 3706 30.435 36 2.73892 2.10192 2014-04-01 15:49:21,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 488 16 3750 3734 30.6022 112 0.889699 2.09043 2014-04-01 15:49:22,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 489 16 3773 3757 30.7277 92 0.722931 2.08158 2014-04-01 15:49:23,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 490 16 3778 3762 30.7058 20 0.534275 2.07971 2014-04-01 15:49:23,614.614 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:24,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:24,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 491 16 3799 3783 30.8143 84 0.468077 2.07385 2014-04-01 15:49:25,113.113 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:25,463.463 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 492 16 3799 3783 30.7517 0 - 2.07385 2014-04-01 15:49:25,874.874 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 451/15522 objects degraded (2.906%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 451, u'num_pgs': 118, u'data_bytes': 21625833828, u'degraded_total': 15522, u'bytes_used': 24884899840, u'op_per_sec': 12, u'write_bytes_sec': 52331479, u'version': 268, u'pgs_by_state': [{u'count': 117, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.906', u'bytes_avail': 2777730535424}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:25,876.876 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:26,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 493 16 3799 3783 30.6893 0 - 2.07385 2014-04-01 15:49:26,632.632 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:27,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 494 16 3799 3783 30.6272 0 - 2.07385 2014-04-01 15:49:28,458.458 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 495 16 3799 3783 30.5653 0 - 2.07385 2014-04-01 15:49:29,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 496 16 3808 3792 30.5762 7.2 5.745 2.08257 2014-04-01 15:49:30,255.255 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:30,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 497 16 3841 3825 30.7803 132 0.596556 2.07799 2014-04-01 15:49:30,976.976 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:31,464.464 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 498 16 3853 3837 30.8148 48 0.13118 2.07408 2014-04-01 15:49:31,667.667 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:32,390.390 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:48:33.205015', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924252, u'health': u'HEALTH_OK', u'kb_used': 9404732, u'store_stats': {u'bytes_total': 10487706, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.895020', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438801092, u'health': u'HEALTH_OK', u'kb_used': 9527892, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:48:33.489364', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438924012, u'health': u'HEALTH_OK', u'kb_used': 9404972, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 430/15738 objects degraded (2.732%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 430, u'num_pgs': 118, u'data_bytes': 21927823716, u'degraded_total': 15738, u'bytes_used': 25329897472, u'op_per_sec': 4, u'write_bytes_sec': 20810974, u'version': 272, u'pgs_by_state': [{u'count': 117, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.732', u'bytes_avail': 2777285537792}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:32,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:32,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 499 16 3872 3856 30.9054 76 0.564905 2.06755 2014-04-01 15:49:33,267.267 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:33,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:49:33.374640min lat: 0.063861 max lat: 135.172 avg lat: 2.0657 2014-04-01 15:49:33,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:49:33,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 500 16 3883 3867 30.9316 44 1.58259 2.0657 2014-04-01 15:49:34,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 501 16 3886 3870 30.8938 12 0.307 2.06456 2014-04-01 15:49:35,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 502 16 3886 3870 30.8322 0 - 2.06456 2014-04-01 15:49:36,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 503 16 3886 3870 30.7709 0 - 2.06456 2014-04-01 15:49:36,958.958 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:37,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 504 16 3896 3880 30.7892 13.3333 4.88744 2.07194 2014-04-01 15:49:38,004.004 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:38,466.466 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 505 16 3921 3905 30.9262 100 0.788879 2.06844 2014-04-01 15:49:38,886.886 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:39,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 506 16 3945 3929 31.0548 96 0.145187 2.05879 2014-04-01 15:49:39,714.714 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:49:33.205298', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855548, u'health': u'HEALTH_OK', u'kb_used': 9473436, u'store_stats': {u'bytes_total': 12584858, u'bytes_log': 12517376, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.895305', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438719064, u'health': u'HEALTH_OK', u'kb_used': 9609920, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.489587', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855180, u'health': u'HEALTH_OK', u'kb_used': 9473804, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 400/15906 objects degraded (2.515%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 25475563, u'degraded_objects': 400, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 22162704740, u'degraded_total': 15906, u'bytes_used': 25918337024, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 49131443, u'op_per_sec': 11, u'version': 276, u'pgs_by_state': [{u'count': 117, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.515', u'bytes_avail': 2776697098240}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:39,715.715 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:40,463.463 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 507 16 3964 3948 31.1435 76 0.535125 2.05236 2014-04-01 15:49:40,525.525 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:41,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 508 16 3968 3952 31.1136 16 0.749862 2.05108 2014-04-01 15:49:42,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 509 16 3968 3952 31.0525 0 - 2.05108 2014-04-01 15:49:43,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 510 16 3968 3952 30.9916 0 - 2.05108 2014-04-01 15:49:44,140.140 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:44,460.460 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 511 16 3968 3952 30.931 0 - 2.05108 2014-04-01 15:49:44,916.916 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:45,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 512 16 3978 3962 30.9487 10 5.80623 2.0563 2014-04-01 15:49:45,551.551 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:46,467.467 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 513 16 4015 3999 31.1768 148 0.328197 2.05151 2014-04-01 15:49:46,538.538 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:49:33.205298', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855548, u'health': u'HEALTH_OK', u'kb_used': 9473436, u'store_stats': {u'bytes_total': 12584858, u'bytes_log': 12517376, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.895305', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438719064, u'health': u'HEALTH_OK', u'kb_used': 9609920, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.489587', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855180, u'health': u'HEALTH_OK', u'kb_used': 9473804, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 392/16188 objects degraded (2.422%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 5634338, u'degraded_objects': 392, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 22556969316, u'degraded_total': 16188, u'bytes_used': 26491453440, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 14649279, u'op_per_sec': 3, u'version': 281, u'pgs_by_state': [{u'count': 117, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.422', u'bytes_avail': 2776123981824}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:46,539.539 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:47,244.244 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:47,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 514 16 4038 4022 31.2951 92 0.486403 2.04338 2014-04-01 15:49:48,460.460 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 515 16 4060 4044 31.4052 88 0.441107 2.03605 2014-04-01 15:49:49,461.461 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 516 16 4062 4046 31.3598 8 0.711648 2.0354 2014-04-01 15:49:50,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 517 16 4062 4046 31.2992 0 - 2.0354 2014-04-01 15:49:50,866.866 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:51,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 518 16 4062 4046 31.2387 0 - 2.0354 2014-04-01 15:49:51,679.679 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:52,307.307 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:49:52,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 519 16 4062 4046 31.1786 0 - 2.0354 2014-04-01 15:49:53,179.179 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:49:33.205298', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855548, u'health': u'HEALTH_OK', u'kb_used': 9473436, u'store_stats': {u'bytes_total': 12584858, u'bytes_log': 12517376, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.895305', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438719064, u'health': u'HEALTH_OK', u'kb_used': 9609920, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.489587', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855180, u'health': u'HEALTH_OK', u'kb_used': 9473804, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 118, u'data_bytes': 22913485156, u'bytes_used': 26875707392, u'op_per_sec': 18, u'version': 285, u'pgs_by_state': [{u'count': 118, u'state_name': u'active+clean'}], u'write_bytes_sec': 77738037, u'bytes_avail': 2775739727872}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 43, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:49:53,180.180 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:49:53,180.180 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:49:53,180.180 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 15:49:53,181.181 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:49:53,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:49:53.377578min lat: 0.063861 max lat: 135.172 avg lat: 2.04442 2014-04-01 15:49:53,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:49:53,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 520 16 4082 4066 31.2724 20 5.38831 2.04442 2014-04-01 15:49:53,931.931 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:49:54,462.462 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 521 16 4108 4092 31.412 104 0.702025 2.03547 2014-04-01 15:49:54,573.573 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 34' 2014-04-01 15:49:55,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 522 16 4138 4122 31.5817 120 0.471935 2.02537 2014-04-01 15:49:56,467.467 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 523 16 4163 4147 31.7125 100 0.615593 2.01704 2014-04-01 15:49:56,607.607 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pgp_num to 34 2014-04-01 15:49:57,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 524 16 4192 4176 31.8733 116 0.798619 2.0067 2014-04-01 15:49:58,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 525 16 4216 4200 31.9954 96 0.326629 1.99951 2014-04-01 15:49:59,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 526 16 4246 4230 32.1627 120 0.378592 1.98893 2014-04-01 15:50:00,463.463 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 527 16 4260 4244 32.2079 56 0.633654 1.98411 2014-04-01 15:50:01,617.617 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:01,618.618 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:01,618.618 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 15:50:01,619.619 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:50:01,619.619 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:50:01,620.620 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 15:50:02,355.355 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 528 16 4260 4244 32.1469 0 - 1.98411 2014-04-01 15:50:02,466.466 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 529 16 4260 4244 32.0861 0 - 1.98411 2014-04-01 15:50:03,463.463 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 530 16 4260 4244 32.0256 0 - 1.98411 2014-04-01 15:50:04,464.464 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 531 16 4260 4244 31.9653 0 - 1.98411 2014-04-01 15:50:05,464.464 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 532 16 4283 4267 32.0781 18.4 0.602463 1.99329 2014-04-01 15:50:06,467.467 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 533 16 4313 4297 32.243 120 0.395322 1.9843 2014-04-01 15:50:07,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 534 16 4338 4322 32.3699 100 0.536224 1.97596 2014-04-01 15:50:07,840.840 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:07,841.841 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:07,841.841 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 15:50:07,842.842 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:50:07,842.842 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:50:07,843.843 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 15:50:08,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 535 16 4365 4349 32.5112 108 0.556751 1.96736 2014-04-01 15:50:09,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 536 16 4400 4384 32.7117 140 0.208902 1.95565 2014-04-01 15:50:10,508.508 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 537 16 4416 4400 32.77 64 0.419166 1.95054 2014-04-01 15:50:11,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 538 16 4418 4402 32.7239 8 0.179267 1.94981 2014-04-01 15:50:12,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 539 16 4428 4412 32.7374 40 0.179466 1.94999 2014-04-01 15:50:13,359.359 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:13,360.360 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:13,360.360 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 15:50:13,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:13,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:50:13.380047min lat: 0.063861 max lat: 135.172 avg lat: 1.94999 2014-04-01 15:50:13,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:50:13,466.466 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 540 16 4428 4412 32.6768 0 - 1.94999 2014-04-01 15:50:14,071.071 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:14,464.464 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 541 16 4433 4417 32.6534 10 0.339319 1.95155 2014-04-01 15:50:14,853.853 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:50:14,854.854 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pg_num 44' 2014-04-01 15:50:15,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 542 16 4438 4422 32.63 20 0.070293 1.9495 2014-04-01 15:50:16,039.039 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pg_num to 44 2014-04-01 15:50:16,465.465 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 543 16 4438 4422 32.5699 0 - 1.9495 2014-04-01 15:50:17,467.467 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 544 16 4438 4422 32.5101 0 - 1.9495 2014-04-01 15:50:18,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 545 16 4468 4452 32.6706 40 0.4247 1.95823 2014-04-01 15:50:19,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 546 16 4492 4476 32.7865 96 1.03872 1.95137 2014-04-01 15:50:20,466.466 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 547 16 4510 4494 32.8582 72 0.435176 1.94578 2014-04-01 15:50:21,048.048 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:21,049.049 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:50:21,050.050 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:21,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 548 16 4533 4517 32.9661 92 0.491158 1.94012 2014-04-01 15:50:21,865.865 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:22,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 549 16 4538 4522 32.9425 20 0.387129 1.93858 2014-04-01 15:50:22,644.644 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:23,293.293 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:23,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 550 16 4538 4522 32.8826 0 - 1.93858 2014-04-01 15:50:23,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:50:24,466.466 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 551 16 4538 4522 32.8229 0 - 1.93858 2014-04-01 15:50:24,680.680 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:49:33.205298', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855548, u'health': u'HEALTH_OK', u'kb_used': 9473436, u'store_stats': {u'bytes_total': 12584858, u'bytes_log': 12517376, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.895305', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438719064, u'health': u'HEALTH_OK', u'kb_used': 9609920, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.489587', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855180, u'health': u'HEALTH_OK', u'kb_used': 9473804, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 13, u'num_pgs': 128, u'recovering_keys_per_sec': 0, u'data_bytes': 24855447964, u'bytes_used': 29170868224, u'recovering_objects_per_sec': 0, u'op_per_sec': 19, u'version': 303, u'pgs_by_state': [{u'count': 1, u'state_name': u'creating'}, {u'count': 3, u'state_name': u'peering'}, {u'count': 124, u'state_name': u'active+clean'}], u'write_bytes_sec': 79557488, u'bytes_avail': 2773444567040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 47, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:50:24,681.681 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:25,385.385 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:25,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 552 16 4538 4522 32.7634 0 - 1.93858 2014-04-01 15:50:26,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 553 16 4548 4532 32.7765 10 5.34329 1.94608 2014-04-01 15:50:27,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 554 16 4577 4561 32.9267 116 0.714155 1.94297 2014-04-01 15:50:28,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 555 16 4601 4585 33.0403 96 0.469254 1.93569 2014-04-01 15:50:29,003.003 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:29,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 556 16 4636 4620 33.2326 140 0.219935 1.92504 2014-04-01 15:50:29,646.646 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:30,432.432 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:50:30,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 557 16 4660 4644 33.3453 96 0.878204 1.91866 2014-04-01 15:50:31,387.387 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.029236', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.089215', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:49:33.205298', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855548, u'health': u'HEALTH_OK', u'kb_used': 9473436, u'store_stats': {u'bytes_total': 12584858, u'bytes_log': 12517376, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.895305', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438719064, u'health': u'HEALTH_OK', u'kb_used': 9609920, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:49:33.489587', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438855180, u'health': u'HEALTH_OK', u'kb_used': 9473804, u'store_stats': {u'bytes_total': 17827738, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 128, u'data_bytes': 24981277084, u'bytes_used': 29728755712, u'op_per_sec': 5, u'version': 306, u'pgs_by_state': [{u'count': 128, u'state_name': u'active+clean'}], u'write_bytes_sec': 23483161, u'bytes_avail': 2772886679552}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 47, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:50:31,388.388 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:50:31,388.388 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:31,389.389 INFO:teuthology.task.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 5, 4, 3] 2014-04-01 15:50:31,389.389 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-04-01 15:50:31,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 558 16 4683 4667 33.4504 92 0.872586 1.91188 2014-04-01 15:50:31,897.897 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:50:31.810812 7f82b4d8b780 0 -- 10.214.133.37:0/1005903 submit_message osd_op(client.4111.0:4699 benchmark_data_plana79_5903_object4698 [write 0~4194304] 3.959f8c5b ack+ondisk+write e47) v4 remote, 10.214.133.37:6800/4936, failed lossy con, dropping message 0x1afccc0 2014-04-01 15:50:32,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 559 16 4712 4696 33.598 116 0.502686 1.90352 2014-04-01 15:50:33,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:50:33.383439min lat: 0.063861 max lat: 135.172 avg lat: 1.89444 2014-04-01 15:50:33,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:50:33,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 560 16 4744 4728 33.7666 128 0.620915 1.89444 2014-04-01 15:50:34,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 561 16 4766 4750 33.8632 88 0.500831 1.88879 2014-04-01 15:50:34,620.620 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.2. 2014-04-01 15:50:35,468.468 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 562 16 4790 4774 33.9738 96 0.126008 1.88149 2014-04-01 15:50:36,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 563 16 4796 4780 33.956 24 1.83741 1.88 2014-04-01 15:50:37,469.469 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 564 16 4798 4782 33.91 8 2.15981 1.8799 2014-04-01 15:50:38,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 565 16 4813 4797 33.9562 60 0.70727 1.87951 2014-04-01 15:50:39,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 566 16 4818 4802 33.9315 20 1.14995 1.87897 2014-04-01 15:50:39,775.775 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:39,776.776 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:39,776.776 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool data 2014-04-01 15:50:39,777.777 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:40,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 567 16 4831 4815 33.9634 52 1.67257 1.87765 2014-04-01 15:50:40,557.557 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:41,447.447 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 24' 2014-04-01 15:50:41,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 568 16 4839 4823 33.9599 32 1.91689 1.88033 2014-04-01 15:50:42,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 569 16 4849 4833 33.9705 40 1.00564 1.87972 2014-04-01 15:50:43,429.429 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pgp_num to 24 2014-04-01 15:50:43,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 570 16 4878 4862 34.1144 116 0.347135 1.87279 2014-04-01 15:50:44,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 571 16 4878 4862 34.0547 0 - 1.87279 2014-04-01 15:50:45,478.478 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 572 16 4896 4880 34.121 36 0.639813 1.87159 2014-04-01 15:50:46,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 573 16 4904 4888 34.1173 32 0.371763 1.87012 2014-04-01 15:50:47,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 574 16 4906 4890 34.0718 8 1.36943 1.86981 2014-04-01 15:50:48,439.439 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:48,440.440 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:48,440.440 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 15:50:48,440.440 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:50:48,441.441 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:50:48,441.441 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 15:50:48,470.470 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 575 16 4907 4891 34.0195 4 2.40803 1.86992 2014-04-01 15:50:49,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 576 16 4907 4891 33.9604 0 - 1.86992 2014-04-01 15:50:50,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 577 16 4915 4899 33.957 16 5.11832 1.87526 2014-04-01 15:50:51,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 578 16 4917 4901 33.9121 8 3.48165 1.8758 2014-04-01 15:50:52,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 579 16 4930 4914 33.9433 52 0.637434 1.87935 2014-04-01 15:50:53,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:50:53.386308min lat: 0.063861 max lat: 135.172 avg lat: 1.87817 2014-04-01 15:50:53,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:50:53,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 580 16 4937 4921 33.933 28 0.183688 1.87817 2014-04-01 15:50:53,942.942 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:50:53,943.943 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:50:53,943.943 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 15:50:53,944.944 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:50:54,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 581 16 4945 4929 33.9297 32 0.346992 1.87895 2014-04-01 15:50:54,675.675 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:50:55,292.292 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:50:55,292.292 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 36' 2014-04-01 15:50:55,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 582 16 4951 4935 33.9126 24 0.444212 1.87768 2014-04-01 15:50:56,385.385 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 36 2014-04-01 15:50:56,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 583 16 4951 4935 33.8545 0 - 1.87768 2014-04-01 15:50:57,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 584 16 4959 4943 33.8513 16 2.47145 1.88003 2014-04-01 15:50:58,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 585 16 4960 4944 33.8003 4 1.05836 1.87986 2014-04-01 15:50:59,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 586 16 4963 4947 33.7631 12 1.66698 1.87976 2014-04-01 15:51:00,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 587 16 4963 4947 33.7056 0 - 1.87976 2014-04-01 15:51:01,400.400 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:51:01,401.401 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 15:51:01,402.402 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:01,471.471 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 588 16 4963 4947 33.6482 0 - 1.87976 2014-04-01 15:51:02,120.120 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:02,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 589 16 4963 4947 33.5911 0 - 1.87976 2014-04-01 15:51:02,903.903 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:03,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 590 16 4963 4947 33.5342 0 - 1.87976 2014-04-01 15:51:03,684.684 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:04,474.474 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:04,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 591 16 4963 4947 33.4774 0 - 1.87976 2014-04-01 15:51:05,193.193 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:50:33.205560', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777176, u'health': u'HEALTH_OK', u'kb_used': 9551808, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.896325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438629136, u'health': u'HEALTH_OK', u'kb_used': 9699848, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.489873', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438776416, u'health': u'HEALTH_OK', u'kb_used': 9552568, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3363/22968 objects degraded (14.642%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 39370853, u'degraded_objects': 3363, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 32027707804, u'degraded_total': 22968, u'bytes_used': 28211191808, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 1031677065, u'op_per_sec': 245, u'version': 323, u'pgs_by_state': [{u'count': 6, u'state_name': u'creating'}, {u'count': 14, u'state_name': u'peering'}, {u'count': 109, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.642', u'bytes_avail': 2307301670912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:05,194.194 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:05,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 592 16 4963 4947 33.4209 0 - 1.87976 2014-04-01 15:51:05,900.900 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:06,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 593 16 4963 4947 33.3645 0 - 1.87976 2014-04-01 15:51:07,472.472 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 594 16 4963 4947 33.3084 0 - 1.87976 2014-04-01 15:51:08,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 595 16 4963 4947 33.2524 0 - 1.87976 2014-04-01 15:51:09,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 596 16 4963 4947 33.1966 0 - 1.87976 2014-04-01 15:51:09,673.673 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:10,386.386 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:10,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 597 16 4963 4947 33.141 0 - 1.87976 2014-04-01 15:51:11,007.007 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:11,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 598 16 4963 4947 33.0856 0 - 1.87976 2014-04-01 15:51:11,782.782 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:50:33.205560', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777176, u'health': u'HEALTH_OK', u'kb_used': 9551808, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.896325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438629136, u'health': u'HEALTH_OK', u'kb_used': 9699848, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.489873', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438776416, u'health': u'HEALTH_OK', u'kb_used': 9552568, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3360/23904 objects degraded (14.056%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3360, u'num_pgs': 138, u'data_bytes': 33336330652, u'degraded_total': 23904, u'bytes_used': 28213243904, u'op_per_sec': 0, u'write_bytes_sec': 838751, u'version': 326, u'pgs_by_state': [{u'count': 5, u'state_name': u'creating'}, {u'count': 15, u'state_name': u'peering'}, {u'count': 109, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.056', u'bytes_avail': 2307299618816}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:11,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:12,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 599 16 4963 4947 33.0303 0 - 1.87976 2014-04-01 15:51:12,489.489 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:13,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:51:13.388934min lat: 0.063861 max lat: 135.172 avg lat: 1.87976 2014-04-01 15:51:13,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:51:13,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 600 16 4963 4947 32.9753 0 - 1.87976 2014-04-01 15:51:14,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 601 16 4963 4947 32.9204 0 - 1.87976 2014-04-01 15:51:15,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 602 16 4963 4947 32.8657 0 - 1.87976 2014-04-01 15:51:16,142.142 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:16,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 603 16 4963 4947 32.8112 0 - 1.87976 2014-04-01 15:51:16,850.850 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:17,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 604 16 4963 4947 32.7569 0 - 1.87976 2014-04-01 15:51:17,622.622 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:18,415.415 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:50:33.205560', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777176, u'health': u'HEALTH_OK', u'kb_used': 9551808, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.896325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438629136, u'health': u'HEALTH_OK', u'kb_used': 9699848, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.489873', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438776416, u'health': u'HEALTH_OK', u'kb_used': 9552568, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'35 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3352/24609 objects degraded (13.621%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8392696, u'degraded_objects': 3352, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 34321992092, u'degraded_total': 24609, u'bytes_used': 28245372928, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 197228377, u'op_per_sec': 47, u'version': 329, u'pgs_by_state': [{u'count': 4, u'state_name': u'creating'}, {u'count': 16, u'state_name': u'peering'}, {u'count': 109, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.621', u'bytes_avail': 2307267489792}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:18,416.416 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:18,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 605 16 4966 4950 32.7226 0.631579 19.4523 1.89039 2014-04-01 15:51:19,232.232 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:19,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 606 16 4967 4951 32.6752 4 0.638165 1.89014 2014-04-01 15:51:20,473.473 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 607 16 4967 4951 32.6213 0 - 1.89014 2014-04-01 15:51:21,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 608 16 4967 4951 32.5677 0 - 1.89014 2014-04-01 15:51:22,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 609 16 4967 4951 32.5142 0 - 1.89014 2014-04-01 15:51:22,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:23,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 610 16 4971 4955 32.4871 4 0.471851 1.89157 2014-04-01 15:51:23,659.659 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:24,432.432 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:24,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 611 16 4974 4958 32.4536 12 0.12185 1.8906 2014-04-01 15:51:25,169.169 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:50:33.205560', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777176, u'health': u'HEALTH_OK', u'kb_used': 9551808, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.896325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438629136, u'health': u'HEALTH_OK', u'kb_used': 9699848, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.489873', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438776416, u'health': u'HEALTH_OK', u'kb_used': 9552568, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'48 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3334/26181 objects degraded (12.734%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 67689347, u'degraded_objects': 3334, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 36519807388, u'degraded_total': 26181, u'bytes_used': 28295761920, u'recovering_objects_per_sec': 16, u'write_bytes_sec': 655583868, u'op_per_sec': 156, u'version': 332, u'pgs_by_state': [{u'count': 2, u'state_name': u'creating'}, {u'count': 16, u'state_name': u'peering'}, {u'count': 111, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.734', u'bytes_avail': 2307217100800}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:25,169.169 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:25,474.474 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 612 16 4974 4958 32.4006 0 - 1.8906 2014-04-01 15:51:25,929.929 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:26,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 613 16 4974 4958 32.3477 0 - 1.8906 2014-04-01 15:51:27,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 614 16 4974 4958 32.295 0 - 1.8906 2014-04-01 15:51:28,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 615 16 4974 4958 32.2425 0 - 1.8906 2014-04-01 15:51:29,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 616 16 4975 4959 32.1967 0.8 30.0066 1.89627 2014-04-01 15:51:29,549.549 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:30,333.333 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:30,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 617 16 4980 4964 32.1769 20 0.492748 1.89606 2014-04-01 15:51:30,960.960 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:31,475.475 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 618 16 4981 4965 32.1313 4 33.7168 1.90247 2014-04-01 15:51:31,712.712 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:50:33.205560', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777176, u'health': u'HEALTH_OK', u'kb_used': 9551808, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.896325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438629136, u'health': u'HEALTH_OK', u'kb_used': 9699848, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:50:33.489873', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438776416, u'health': u'HEALTH_OK', u'kb_used': 9552568, u'store_stats': {u'bytes_total': 20973466, u'bytes_log': 20905984, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3283/27624 objects degraded (11.885%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14251065, u'degraded_objects': 3283, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 38537267612, u'degraded_total': 27624, u'bytes_used': 28463398912, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 846156989, u'op_per_sec': 201, u'version': 335, u'pgs_by_state': [{u'count': 14, u'state_name': u'peering'}, {u'count': 115, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.885', u'bytes_avail': 2307049463808}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:31,713.713 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:32,413.413 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:32,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 619 16 4981 4965 32.0794 0 - 1.90247 2014-04-01 15:51:33,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:51:33.391916min lat: 0.063861 max lat: 135.172 avg lat: 1.90247 2014-04-01 15:51:33,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:51:33,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 620 16 4981 4965 32.0277 0 - 1.90247 2014-04-01 15:51:34,476.476 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 621 16 4986 4970 32.0083 6.66667 39.5795 1.94 2014-04-01 15:51:35,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 622 16 4988 4972 31.9697 8 40.4397 1.95543 2014-04-01 15:51:36,053.053 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:36,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 623 16 4995 4979 31.9633 28 13.6122 1.97422 2014-04-01 15:51:36,762.762 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:37,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 624 16 5009 4993 32.0018 56 0.771951 1.98471 2014-04-01 15:51:37,769.769 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:38,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 625 16 5018 5002 32.0082 36 1.76596 1.98361 2014-04-01 15:51:38,625.625 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4074/27675 objects degraded (14.721%); 142/9235 unfound (1.538%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8345026, u'degraded_objects': 4074, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 38608570780, u'degraded_total': 27675, u'bytes_used': 28830019584, u'unfound_ratio': u'1.538', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 15762827, u'op_per_sec': 3, u'unfound_objects': 142, u'version': 339, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 123, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.721', u'bytes_avail': 2306682843136, u'unfound_total': 9235}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:38,626.626 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:39,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:39,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 626 16 5023 5007 31.989 20 1.16389 1.9835 2014-04-01 15:51:40,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 627 16 5025 5009 31.9508 8 2.01063 1.98353 2014-04-01 15:51:41,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 628 16 5030 5014 31.9317 20 2.71502 1.98327 2014-04-01 15:51:42,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 629 16 5031 5015 31.8873 4 0.283198 1.98293 2014-04-01 15:51:43,058.058 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:43,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 630 16 5035 5019 31.8621 16 6.52501 1.98539 2014-04-01 15:51:43,765.765 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:44,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 631 16 5037 5021 31.8243 8 5.67353 1.98585 2014-04-01 15:51:44,550.550 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:45,437.437 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4035/27795 objects degraded (14.517%); 142/9275 unfound (1.531%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 4035, u'num_pgs': 138, u'data_bytes': 38776342940, u'degraded_total': 27795, u'bytes_used': 29112565760, u'unfound_ratio': u'1.531', u'op_per_sec': 7, u'write_bytes_sec': 31542257, u'unfound_objects': 142, u'version': 343, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 123, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.517', u'bytes_avail': 2306400296960, u'unfound_total': 9275}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:45,437.437 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:45,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 632 16 5040 5024 31.7929 12 7.54889 1.98865 2014-04-01 15:51:46,138.138 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:46,478.478 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 633 16 5042 5026 31.7553 8 0.439116 1.98934 2014-04-01 15:51:47,477.477 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 634 16 5042 5026 31.7052 0 - 1.98934 2014-04-01 15:51:48,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 635 16 5042 5026 31.6553 0 - 1.98934 2014-04-01 15:51:49,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 636 16 5042 5026 31.6055 0 - 1.98934 2014-04-01 15:51:49,894.894 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:50,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 637 16 5042 5026 31.5559 0 - 1.98934 2014-04-01 15:51:50,773.773 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:51,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 638 16 5050 5034 31.5566 6.4 7.54513 2.00063 2014-04-01 15:51:51,769.769 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:52,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 639 16 5059 5043 31.5636 36 1.6358 2.00699 2014-04-01 15:51:52,504.504 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3983/27825 objects degraded (14.314%); 142/9285 unfound (1.529%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3983, u'num_pgs': 138, u'data_bytes': 38818285980, u'degraded_total': 27825, u'bytes_used': 29380808704, u'unfound_ratio': u'1.529', u'op_per_sec': 0, u'write_bytes_sec': 2928619, u'unfound_objects': 142, u'version': 346, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 123, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.314', u'bytes_avail': 2306132054016, u'unfound_total': 9285}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:52,504.504 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:53,264.264 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:53,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:51:53.394415min lat: 0.063861 max lat: 135.172 avg lat: 2.0078 2014-04-01 15:51:53,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:51:53,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 640 16 5062 5046 31.533 12 1.00736 2.0078 2014-04-01 15:51:54,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 641 16 5067 5051 31.515 20 3.34623 2.00953 2014-04-01 15:51:55,478.478 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 642 16 5067 5051 31.4659 0 - 2.00953 2014-04-01 15:51:56,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 643 16 5077 5061 31.4792 20 4.33879 2.01255 2014-04-01 15:51:56,921.921 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:57,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 644 16 5086 5070 31.4862 36 1.04582 2.01336 2014-04-01 15:51:57,701.701 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:51:58,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:51:58,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 645 16 5089 5073 31.456 12 1.48599 2.01299 2014-04-01 15:51:59,218.218 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3933/27918 objects degraded (14.088%); 142/9316 unfound (1.524%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5186273, u'degraded_objects': 3933, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 38948309404, u'degraded_total': 27918, u'bytes_used': 29673431040, u'unfound_ratio': u'1.524', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 29821072, u'op_per_sec': 7, u'unfound_objects': 142, u'version': 350, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.088', u'bytes_avail': 2305839431680, u'unfound_total': 9316}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:51:59,219.219 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:51:59,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 646 16 5090 5074 31.4135 4 1.96526 2.01298 2014-04-01 15:51:59,964.964 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:00,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 647 16 5099 5083 31.4205 36 0.260695 2.01262 2014-04-01 15:52:01,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 648 16 5104 5088 31.4029 20 1.13299 2.01211 2014-04-01 15:52:02,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 649 16 5106 5090 31.3669 8 5.61575 2.01349 2014-04-01 15:52:03,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 650 16 5108 5092 31.3309 8 7.08342 2.01452 2014-04-01 15:52:03,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:04,318.318 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:04,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 651 16 5109 5093 31.2889 4 6.11571 2.01532 2014-04-01 15:52:05,103.103 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:05,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 652 16 5109 5093 31.2409 0 - 2.01532 2014-04-01 15:52:05,979.979 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3902/28011 objects degraded (13.930%); 142/9347 unfound (1.519%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11264674, u'degraded_objects': 3902, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39078332828, u'degraded_total': 28011, u'bytes_used': 30004518912, u'unfound_ratio': u'1.519', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 23780979, u'op_per_sec': 5, u'unfound_objects': 142, u'version': 354, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.930', u'bytes_avail': 2305508343808, u'unfound_total': 9347}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:05,979.979 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:06,481.481 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 653 16 5113 5097 31.2176 8 1.92355 2.01748 2014-04-01 15:52:06,696.696 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:07,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 654 16 5119 5103 31.2066 24 5.38952 2.02188 2014-04-01 15:52:08,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 655 16 5120 5104 31.165 4 0.891109 2.02166 2014-04-01 15:52:09,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 656 16 5122 5106 31.1297 8 10.897 2.02319 2014-04-01 15:52:10,312.312 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:10,480.480 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 657 16 5126 5110 31.1067 16 1.73741 2.02623 2014-04-01 15:52:10,901.901 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:11,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 658 16 5126 5110 31.0594 0 - 2.02623 2014-04-01 15:52:11,843.843 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:12,481.481 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 659 16 5127 5111 31.0183 2 11.3074 2.02805 2014-04-01 15:52:12,562.562 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3847/28059 objects degraded (13.710%); 142/9363 unfound (1.517%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10242428, u'degraded_objects': 3847, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39145441692, u'degraded_total': 28059, u'bytes_used': 30403584000, u'unfound_ratio': u'1.517', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 9104380, u'op_per_sec': 2, u'unfound_objects': 142, u'version': 358, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.710', u'bytes_avail': 2305109278720, u'unfound_total': 9363}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:12,563.563 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:13,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:52:13.396978min lat: 0.063861 max lat: 135.172 avg lat: 2.03142 2014-04-01 15:52:13,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:52:13,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 660 16 5131 5115 30.9956 16 7.63678 2.03142 2014-04-01 15:52:14,481.481 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 661 16 5144 5128 31.0273 52 4.06426 2.04057 2014-04-01 15:52:15,481.481 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 662 16 5145 5129 30.9865 4 1.08242 2.04039 2014-04-01 15:52:16,406.406 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:16,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 663 16 5147 5131 30.9518 8 5.80168 2.04118 2014-04-01 15:52:17,482.482 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 664 16 5147 5131 30.9052 0 - 2.04118 2014-04-01 15:52:18,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 665 16 5154 5138 30.9009 14 4.57012 2.04351 2014-04-01 15:52:19,481.481 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 666 16 5158 5142 30.8785 16 0.166504 2.04365 2014-04-01 15:52:20,023.023 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:20,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 667 16 5159 5143 30.8382 4 1.11621 2.04347 2014-04-01 15:52:20,627.627 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:21,482.482 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 668 16 5181 5165 30.9237 88 0.581012 2.04749 2014-04-01 15:52:21,582.582 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:22,297.297 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3777/28161 objects degraded (13.412%); 142/9397 unfound (1.511%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3777, u'num_pgs': 138, u'data_bytes': 39288048028, u'degraded_total': 28161, u'bytes_used': 30714859520, u'unfound_ratio': u'1.511', u'op_per_sec': 3, u'write_bytes_sec': 13940250, u'unfound_objects': 142, u'version': 364, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.412', u'bytes_avail': 2304798003200, u'unfound_total': 9397}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:22,297.297 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:22,482.482 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 669 16 5184 5168 30.8954 12 0.926403 2.04694 2014-04-01 15:52:23,011.011 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:23,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 670 16 5186 5170 30.8613 8 1.07441 2.04654 2014-04-01 15:52:24,482.482 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 671 16 5189 5173 30.8332 12 3.33604 2.04727 2014-04-01 15:52:25,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 672 16 5189 5173 30.7873 0 - 2.04727 2014-04-01 15:52:26,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 673 16 5190 5174 30.7475 2 5.11259 2.04786 2014-04-01 15:52:26,628.628 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:27,058.058 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:52:26.973906 7f3448208700 0 -- :/1014564 >> 10.214.133.37:6790/0 pipe(0x7f34440193f0 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3444019650).fault 2014-04-01 15:52:27,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:27,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 674 16 5196 5180 30.7375 24 6.42717 2.05128 2014-04-01 15:52:28,145.145 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:28,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 675 16 5196 5180 30.6919 0 - 2.05128 2014-04-01 15:52:28,866.866 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:51:33.205829', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438726512, u'health': u'HEALTH_OK', u'kb_used': 9602472, u'store_stats': {u'bytes_total': 15730586, u'bytes_log': 15663104, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.919133', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438567460, u'health': u'HEALTH_OK', u'kb_used': 9761524, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 06:51:33.490086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438725920, u'health': u'HEALTH_OK', u'kb_used': 9603064, u'store_stats': {u'bytes_total': 23070618, u'bytes_log': 23003136, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3735/28269 objects degraded (13.212%); 142/9433 unfound (1.505%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6443300, u'degraded_objects': 3735, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39439042972, u'degraded_total': 28269, u'bytes_used': 31136169984, u'unfound_ratio': u'1.505', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 10309281, u'op_per_sec': 2, u'unfound_objects': 142, u'version': 368, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.212', u'bytes_avail': 2304376692736, u'unfound_total': 9433}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:28,867.867 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:29,483.483 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 676 16 5196 5180 30.6465 0 - 2.05128 2014-04-01 15:52:29,553.553 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:52:29.469809 7fc48a7fc700 0 monclient: hunting for new mon 2014-04-01 15:52:29,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:30,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 677 16 5196 5180 30.6012 0 - 2.05128 2014-04-01 15:52:31,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 678 16 5199 5183 30.5738 3 0.568634 2.05365 2014-04-01 15:52:32,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 679 16 5202 5186 30.5465 12 8.45729 2.05617 2014-04-01 15:52:33,347.347 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:33,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:52:33.399942min lat: 0.063861 max lat: 135.172 avg lat: 2.05856 2014-04-01 15:52:33,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:52:33,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 680 16 5205 5189 30.5192 12 1.67548 2.05856 2014-04-01 15:52:33,776.776 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:52:33.692397 7f11358ec700 0 monclient: hunting for new mon 2014-04-01 15:52:33,909.909 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:34,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 681 16 5205 5189 30.4744 0 - 2.05856 2014-04-01 15:52:34,835.835 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:35,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 682 16 5205 5189 30.4297 0 - 2.05856 2014-04-01 15:52:35,554.554 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3671/28296 objects degraded (12.974%); 142/9442 unfound (1.504%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12319484, u'degraded_objects': 3671, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39476791708, u'degraded_total': 28296, u'bytes_used': 31362244608, u'unfound_ratio': u'1.504', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 7391690, u'op_per_sec': 1, u'unfound_objects': 142, u'version': 372, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.974', u'bytes_avail': 2304150618112, u'unfound_total': 9442}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:35,555.555 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:36,484.484 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 683 16 5213 5197 30.432 10.6667 0.183063 2.06175 2014-04-01 15:52:36,529.529 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:37,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 684 16 5213 5197 30.3875 0 - 2.06175 2014-04-01 15:52:38,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 685 16 5213 5197 30.3431 0 - 2.06175 2014-04-01 15:52:39,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 686 16 5218 5202 30.328 6.66667 0.192674 2.06691 2014-04-01 15:52:40,144.144 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:40,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 687 16 5218 5202 30.2839 0 - 2.06691 2014-04-01 15:52:40,978.978 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:41,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 688 16 5222 5206 30.2631 8 2.4617 2.07193 2014-04-01 15:52:41,615.615 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:42,421.421 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3618/28341 objects degraded (12.766%); 142/9457 unfound (1.502%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3618, u'num_pgs': 138, u'data_bytes': 39539706268, u'degraded_total': 28341, u'bytes_used': 31594606592, u'unfound_ratio': u'1.502', u'op_per_sec': 0, u'write_bytes_sec': 986415, u'unfound_objects': 142, u'version': 376, u'pgs_by_state': [{u'count': 123, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.766', u'bytes_avail': 2303918256128, u'unfound_total': 9457}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:42,421.421 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:42,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 689 16 5223 5207 30.225 4 3.44355 2.0722 2014-04-01 15:52:43,279.279 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:43,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 690 16 5227 5211 30.2044 16 2.29081 2.07674 2014-04-01 15:52:44,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 691 16 5227 5211 30.1607 0 - 2.07674 2014-04-01 15:52:45,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 692 16 5229 5213 30.1287 4 18.2211 2.08134 2014-04-01 15:52:46,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 693 16 5233 5217 30.1083 16 0.660521 2.08609 2014-04-01 15:52:46,904.904 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:47,485.485 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 694 16 5234 5218 30.0706 4 4.18443 2.08649 2014-04-01 15:52:47,669.669 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:48,442.442 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:48,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 695 16 5234 5218 30.0274 0 - 2.08649 2014-04-01 15:52:49,162.162 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3072/28392 objects degraded (10.820%); 142/9474 unfound (1.499%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6460776, u'degraded_objects': 3072, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39611009436, u'degraded_total': 28392, u'bytes_used': 31915958272, u'unfound_ratio': u'1.499', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 11629398, u'op_per_sec': 2, u'unfound_objects': 142, u'version': 380, u'pgs_by_state': [{u'count': 125, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.820', u'bytes_avail': 2303596904448, u'unfound_total': 9474}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:49,162.162 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:49,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 696 16 5237 5221 30.0015 6 17.0414 2.09159 2014-04-01 15:52:49,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:50,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 697 16 5250 5234 30.033 52 0.198165 2.10115 2014-04-01 15:52:51,486.486 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 698 16 5257 5241 30.0301 28 1.31915 2.10109 2014-04-01 15:52:52,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 699 16 5270 5254 30.0615 52 0.642819 2.10035 2014-04-01 15:52:53,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:52:53.402666min lat: 0.063861 max lat: 135.172 avg lat: 2.09855 2014-04-01 15:52:53,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:52:53,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 700 16 5278 5262 30.0643 32 0.552736 2.09855 2014-04-01 15:52:53,507.507 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:54,067.067 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:54,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 701 16 5278 5262 30.0214 0 - 2.09855 2014-04-01 15:52:54,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:52:55,487.487 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 702 16 5278 5262 29.9786 0 - 2.09855 2014-04-01 15:52:55,563.563 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2529/28476 objects degraded (8.881%); 142/9502 unfound (1.494%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7562443, u'degraded_objects': 2529, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 39728449948, u'degraded_total': 28476, u'bytes_used': 32020783104, u'unfound_ratio': u'1.494', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 21426924, u'op_per_sec': 5, u'unfound_objects': 142, u'version': 384, u'pgs_by_state': [{u'count': 126, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.881', u'bytes_avail': 2303492079616, u'unfound_total': 9502}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:52:55,563.563 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:52:56,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:52:56,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 703 16 5278 5262 29.936 0 - 2.09855 2014-04-01 15:52:57,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 704 16 5278 5262 29.8935 0 - 2.09855 2014-04-01 15:52:58,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 705 16 5278 5262 29.8511 0 - 2.09855 2014-04-01 15:52:59,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 706 16 5285 5269 29.8484 4.66667 6.94275 2.10529 2014-04-01 15:52:59,978.978 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:00,492.492 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 707 16 5308 5292 29.9363 92 0.42499 2.10869 2014-04-01 15:53:00,626.626 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:01,462.462 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:01,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 708 16 5317 5301 29.9449 36 0.355056 2.10626 2014-04-01 15:53:02,189.189 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2277/28548 objects degraded (7.976%); 142/9526 unfound (1.491%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2277, u'num_pgs': 138, u'data_bytes': 39829113244, u'degraded_total': 28548, u'bytes_used': 32087314432, u'unfound_ratio': u'1.491', u'op_per_sec': 2, u'write_bytes_sec': 11002870, u'unfound_objects': 142, u'version': 388, u'pgs_by_state': [{u'count': 127, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.976', u'bytes_avail': 2303425548288, u'unfound_total': 9526}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:02,189.189 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:02,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 709 16 5324 5308 29.9421 28 0.634842 2.10467 2014-04-01 15:53:02,996.996 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:03,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 710 16 5326 5310 29.9112 8 0.88765 2.10449 2014-04-01 15:53:04,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 711 16 5329 5313 29.886 12 0.093306 2.10457 2014-04-01 15:53:05,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 712 16 5339 5323 29.9002 40 0.128006 2.10402 2014-04-01 15:53:06,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 713 16 5345 5329 29.892 24 0.112428 2.10297 2014-04-01 15:53:06,641.641 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:07,359.359 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:07,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 714 16 5352 5336 29.8893 28 0.130781 2.10312 2014-04-01 15:53:07,992.992 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:08,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 715 16 5355 5339 29.8643 12 2.06963 2.10413 2014-04-01 15:53:08,726.726 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2011/28719 objects degraded (7.002%); 142/9583 unfound (1.482%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8586205, u'degraded_objects': 2011, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 40068188572, u'degraded_total': 28719, u'bytes_used': 32455335936, u'unfound_ratio': u'1.482', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 29574706, u'op_per_sec': 7, u'unfound_objects': 142, u'version': 391, u'pgs_by_state': [{u'count': 128, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.002', u'bytes_avail': 2303057526784, u'unfound_total': 9583}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:08,726.726 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:09,488.488 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 716 16 5357 5341 29.8337 8 0.181171 2.10409 2014-04-01 15:53:09,516.516 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:10,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 717 16 5359 5343 29.8033 8 2.59016 2.10559 2014-04-01 15:53:11,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 718 16 5359 5343 29.7618 0 - 2.10559 2014-04-01 15:53:12,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 719 16 5359 5343 29.7204 0 - 2.10559 2014-04-01 15:53:13,132.132 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:13,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:53:13.405572min lat: 0.063861 max lat: 135.172 avg lat: 2.10559 2014-04-01 15:53:13,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:53:13,489.489 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 720 16 5359 5343 29.6791 0 - 2.10559 2014-04-01 15:53:13,837.837 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:14,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 721 16 5359 5343 29.6379 0 - 2.10559 2014-04-01 15:53:14,606.606 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:15,328.328 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1990/28758 objects degraded (6.920%); 142/9596 unfound (1.480%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4980184, u'degraded_objects': 1990, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 40122714524, u'degraded_total': 28758, u'bytes_used': 32626200576, u'unfound_ratio': u'1.480', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 8715323, u'op_per_sec': 2, u'unfound_objects': 142, u'version': 395, u'pgs_by_state': [{u'count': 128, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.920', u'bytes_avail': 2302886662144, u'unfound_total': 9596}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:15,328.328 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:15,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 722 16 5362 5346 29.6135 2.4 13.8342 2.10843 2014-04-01 15:53:16,038.038 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:16,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 723 16 5364 5348 29.5836 8 0.072763 2.10771 2014-04-01 15:53:17,490.490 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 724 16 5364 5348 29.5427 0 - 2.10771 2014-04-01 15:53:18,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 725 16 5364 5348 29.502 0 - 2.10771 2014-04-01 15:53:19,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 726 16 5364 5348 29.4614 0 - 2.10771 2014-04-01 15:53:19,651.651 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:20,242.242 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:20,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 727 16 5372 5356 29.4648 8 18.1433 2.11431 2014-04-01 15:53:21,038.038 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:21,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 728 16 5381 5365 29.4738 36 0.175307 2.12013 2014-04-01 15:53:21,863.863 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1970/28779 objects degraded (6.845%); 142/9603 unfound (1.479%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1970, u'num_pgs': 138, u'data_bytes': 40152074652, u'degraded_total': 28779, u'bytes_used': 32812449792, u'unfound_ratio': u'1.479', u'op_per_sec': 0, u'write_bytes_sec': 3699750, u'unfound_objects': 142, u'version': 399, u'pgs_by_state': [{u'count': 128, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.845', u'bytes_avail': 2302700412928, u'unfound_total': 9603}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:21,864.864 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:22,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 729 16 5381 5365 29.4334 0 - 2.12013 2014-04-01 15:53:22,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:23,492.492 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 730 16 5381 5365 29.3931 0 - 2.12013 2014-04-01 15:53:24,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 731 16 5400 5384 29.4568 25.3333 0.476314 2.14032 2014-04-01 15:53:25,491.491 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 732 16 5411 5395 29.4767 44 0.881816 2.13727 2014-04-01 15:53:26,253.253 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:26,492.492 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 733 16 5424 5408 29.5074 52 2.08841 2.13442 2014-04-01 15:53:26,826.826 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:27,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 734 16 5438 5422 29.5435 56 1.76091 2.13296 2014-04-01 15:53:27,769.769 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:28,490.490 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:52:33.206100', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438699188, u'health': u'HEALTH_OK', u'kb_used': 9629796, u'store_stats': {u'bytes_total': 3171945, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:52:33.948989', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438501380, u'health': u'HEALTH_OK', u'kb_used': 9827604, u'store_stats': {u'bytes_total': 3172591, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:52:33.490376', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438698920, u'health': u'HEALTH_OK', u'kb_used': 9630064, u'store_stats': {u'bytes_total': 3171889, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1912/28947 objects degraded (6.605%); 142/9659 unfound (1.470%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2571249, u'degraded_objects': 1912, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 40386955676, u'degraded_total': 28947, u'bytes_used': 33099456512, u'unfound_ratio': u'1.470', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 53996233, u'op_per_sec': 12, u'unfound_objects': 142, u'version': 403, u'pgs_by_state': [{u'count': 129, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.605', u'bytes_avail': 2302413406208, u'unfound_total': 9659}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:28,491.491 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:28,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 735 16 5443 5427 29.5305 20 0.953704 2.13184 2014-04-01 15:53:29,187.187 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:29,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 736 16 5443 5427 29.4904 0 - 2.13184 2014-04-01 15:53:30,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 737 16 5443 5427 29.4503 0 - 2.13184 2014-04-01 15:53:31,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 738 16 5446 5430 29.4267 4 5.81582 2.13356 2014-04-01 15:53:32,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 739 16 5451 5435 29.4139 20 0.714516 2.13352 2014-04-01 15:53:32,814.814 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:33,560.560 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:33,762.762 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:53:33.408597min lat: 0.063861 max lat: 135.172 avg lat: 2.13647 2014-04-01 15:53:33,762.762 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:53:33,763.763 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 740 16 5470 5454 29.4769 76 0.602023 2.13647 2014-04-01 15:53:34,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 741 16 5484 5468 29.5127 56 0.18719 2.13297 2014-04-01 15:53:34,630.630 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:34,653.653 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:53:34.568240 7f829a7f3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a000c3d0 sd=8 :33417 s=2 pgs=11 cs=1 l=1 c=0x7f82a000ce50).injecting socket failure 2014-04-01 15:53:35,423.423 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1917/29049 objects degraded (6.599%); 142/9693 unfound (1.465%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1917, u'num_pgs': 138, u'data_bytes': 40529562012, u'degraded_total': 29049, u'bytes_used': 33509306368, u'unfound_ratio': u'1.465', u'op_per_sec': 6, u'write_bytes_sec': 28526731, u'unfound_objects': 142, u'version': 407, u'pgs_by_state': [{u'count': 129, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.599', u'bytes_avail': 2302003556352, u'unfound_total': 9693}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:35,424.424 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:35,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 742 16 5489 5473 29.4998 20 0.1142 2.13183 2014-04-01 15:53:36,343.343 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:36,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 743 16 5495 5479 29.4924 24 0.461861 2.13071 2014-04-01 15:53:37,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 744 16 5501 5485 29.485 24 1.46354 2.13195 2014-04-01 15:53:38,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 745 16 5508 5492 29.483 28 3.88745 2.1336 2014-04-01 15:53:39,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 746 16 5513 5497 29.4703 20 0.125554 2.13273 2014-04-01 15:53:40,261.261 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:40,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 747 16 5517 5501 29.4523 16 2.81476 2.13324 2014-04-01 15:53:41,039.039 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:41,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 748 16 5533 5517 29.4985 64 0.405607 2.13412 2014-04-01 15:53:41,833.833 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:42,493.493 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 749 16 5539 5523 29.4911 24 1.72046 2.13299 2014-04-01 15:53:42,553.553 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1907/29205 objects degraded (6.530%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1907, u'num_pgs': 138, u'data_bytes': 40747665820, u'degraded_total': 29205, u'bytes_used': 33788571648, u'op_per_sec': 3, u'write_bytes_sec': 14651963, u'version': 411, u'pgs_by_state': [{u'count': 129, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.530', u'bytes_avail': 2301724291072}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:42,553.553 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:43,341.341 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:43,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 750 16 5552 5536 29.5211 52 1.36456 2.13116 2014-04-01 15:53:44,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 751 16 5557 5541 29.5084 20 1.40565 2.13097 2014-04-01 15:53:45,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 752 16 5569 5553 29.533 48 1.27305 2.12942 2014-04-01 15:53:46,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 753 16 5569 5553 29.4938 0 - 2.12942 2014-04-01 15:53:46,984.984 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:47,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 754 16 5571 5555 29.4653 4 2.20934 2.1294 2014-04-01 15:53:47,691.691 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:48,470.470 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:48,494.494 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 755 16 5585 5569 29.5004 56 0.69487 2.13224 2014-04-01 15:53:49,215.215 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1669/29397 objects degraded (5.677%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1669, u'num_pgs': 138, u'data_bytes': 41016101276, u'degraded_total': 29397, u'bytes_used': 34293989376, u'op_per_sec': 8, u'write_bytes_sec': 35422261, u'version': 414, u'pgs_by_state': [{u'count': 130, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.677', u'bytes_avail': 2301218873344}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:49,216.216 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:49,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 756 16 5603 5587 29.5566 72 0.199567 2.12862 2014-04-01 15:53:49,918.918 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:50,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 757 16 5613 5597 29.5704 40 0.504253 2.12657 2014-04-01 15:53:51,498.498 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 758 16 5621 5605 29.5736 32 0.110299 2.12607 2014-04-01 15:53:52,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 759 16 5629 5613 29.5768 32 0.258082 2.12426 2014-04-01 15:53:53,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:53:53.411448min lat: 0.063861 max lat: 135.172 avg lat: 2.12387 2014-04-01 15:53:53,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:53:53,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 760 16 5635 5619 29.5695 24 0.631049 2.12387 2014-04-01 15:53:53,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:54,153.153 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:54,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 761 16 5641 5625 29.5621 24 0.42249 2.1236 2014-04-01 15:53:55,111.111 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:53:55,495.495 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 762 16 5646 5630 29.5496 20 0.191125 2.12194 2014-04-01 15:53:55,877.877 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1653/29574 objects degraded (5.589%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7453027, u'degraded_objects': 1653, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 41263565212, u'degraded_total': 29574, u'bytes_used': 34766274560, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 48444675, u'op_per_sec': 11, u'version': 418, u'pgs_by_state': [{u'count': 130, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.589', u'bytes_avail': 2300746588160}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:53:55,878.878 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:53:56,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 763 16 5649 5633 29.5266 12 4.48918 2.12293 2014-04-01 15:53:56,574.574 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:53:57,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 764 16 5649 5633 29.4879 0 - 2.12293 2014-04-01 15:53:58,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 765 16 5651 5635 29.4598 4 6.55526 2.12461 2014-04-01 15:53:59,496.496 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 766 16 5652 5636 29.4266 4 3.59077 2.12487 2014-04-01 15:54:00,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:00,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 767 16 5664 5648 29.4508 48 0.38859 2.12966 2014-04-01 15:54:00,759.759 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:01,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 768 16 5668 5652 29.4333 16 0.217073 2.13197 2014-04-01 15:54:01,685.685 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:02,405.405 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1638/29643 objects degraded (5.526%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1638, u'num_pgs': 138, u'data_bytes': 41360034204, u'degraded_total': 29643, u'bytes_used': 34953654272, u'op_per_sec': 1, u'write_bytes_sec': 6349772, u'version': 422, u'pgs_by_state': [{u'count': 130, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.526', u'bytes_avail': 2300559208448}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:02,405.405 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:02,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 769 16 5668 5652 29.395 0 - 2.13197 2014-04-01 15:54:03,098.098 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:03,497.497 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 770 16 5668 5652 29.3568 0 - 2.13197 2014-04-01 15:54:04,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 771 16 5668 5652 29.3188 0 - 2.13197 2014-04-01 15:54:05,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 772 16 5668 5652 29.2808 0 - 2.13197 2014-04-01 15:54:06,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 773 16 5668 5652 29.2429 0 - 2.13197 2014-04-01 15:54:06,712.712 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:07,529.529 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:07,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 774 16 5669 5653 29.2103 0.666667 11.0152 2.13354 2014-04-01 15:54:08,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:08,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 775 16 5669 5653 29.1726 0 - 2.13354 2014-04-01 15:54:09,128.128 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1607/29685 objects degraded (5.414%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3841565, u'degraded_objects': 1607, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 41418754460, u'degraded_total': 29685, u'bytes_used': 35157913600, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 3841565, u'op_per_sec': 0, u'version': 425, u'pgs_by_state': [{u'count': 130, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.414', u'bytes_avail': 2300354949120}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:09,129.129 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:09,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 776 16 5669 5653 29.135 0 - 2.13354 2014-04-01 15:54:09,908.908 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:10,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 777 16 5674 5658 29.1232 6.66667 0.217028 2.13967 2014-04-01 15:54:11,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 778 16 5675 5659 29.0909 4 0.105958 2.13931 2014-04-01 15:54:12,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 779 16 5675 5659 29.0536 0 - 2.13931 2014-04-01 15:54:13,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:54:13.415022min lat: 0.063861 max lat: 135.172 avg lat: 2.13931 2014-04-01 15:54:13,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:54:13,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 780 16 5675 5659 29.0163 0 - 2.13931 2014-04-01 15:54:13,787.787 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:14,442.442 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:14,757.757 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 781 16 5675 5659 28.9792 0 - 2.13931 2014-04-01 15:54:15,369.369 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:15,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 782 16 5680 5664 28.9677 5 5.10032 2.14764 2014-04-01 15:54:16,307.307 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1270/29709 objects degraded (4.275%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6474844, u'degraded_objects': 1270, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 41452308892, u'degraded_total': 29709, u'bytes_used': 35271892992, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 6474844, u'op_per_sec': 1, u'version': 429, u'pgs_by_state': [{u'count': 131, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.275', u'bytes_avail': 2300240969728}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:16,308.308 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:16,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 783 16 5706 5690 29.0635 104 0.340118 2.16191 2014-04-01 15:54:17,001.001 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:17,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 784 16 5719 5703 29.0928 52 0.522239 2.15933 2014-04-01 15:54:18,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 785 16 5725 5709 29.0863 24 2.1988 2.1577 2014-04-01 15:54:19,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 786 16 5732 5716 29.0849 28 1.45879 2.15681 2014-04-01 15:54:20,500.500 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 787 16 5738 5722 29.0784 24 1.88097 2.15627 2014-04-01 15:54:20,641.641 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:21,350.350 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:21,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 788 16 5738 5722 29.0415 0 - 2.15627 2014-04-01 15:54:21,974.974 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:22,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 789 16 5738 5722 29.0047 0 - 2.15627 2014-04-01 15:54:22,749.749 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1016/29904 objects degraded (3.398%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9023429, u'degraded_objects': 1016, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 41724938652, u'degraded_total': 29904, u'bytes_used': 35843784704, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 36093718, u'op_per_sec': 8, u'version': 433, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.398', u'bytes_avail': 2299669078016}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:22,749.749 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:23,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 790 16 5747 5731 29.0135 12 7.20785 2.16157 2014-04-01 15:54:23,603.603 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:24,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 791 16 5762 5746 29.0527 60 1.14504 2.16103 2014-04-01 15:54:25,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 792 16 5780 5764 29.1069 72 0.947782 2.15658 2014-04-01 15:54:26,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 793 16 5794 5778 29.1408 56 1.82729 2.15481 2014-04-01 15:54:27,217.217 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:27,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 794 16 5797 5781 29.1192 12 1.03422 2.15416 2014-04-01 15:54:28,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:28,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 795 16 5799 5783 29.0927 8 0.113502 2.15363 2014-04-01 15:54:28,825.825 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:29,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 796 16 5799 5783 29.0561 0 - 2.15363 2014-04-01 15:54:29,604.604 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:53:33.206385', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646632, u'health': u'HEALTH_OK', u'kb_used': 9682352, u'store_stats': {u'bytes_total': 5269097, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:53:33.949194', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438436356, u'health': u'HEALTH_OK', u'kb_used': 9892628, u'store_stats': {u'bytes_total': 6318319, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:53:33.490666', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438646588, u'health': u'HEALTH_OK', u'kb_used': 9682396, u'store_stats': {u'bytes_total': 6317617, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1016/30051 objects degraded (3.381%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5151147, u'degraded_objects': 1016, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 41930459548, u'degraded_total': 30051, u'bytes_used': 36056694784, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 55374837, u'op_per_sec': 13, u'version': 436, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.381', u'bytes_avail': 2299456167936}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:29,604.604 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:30,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:30,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 797 16 5799 5783 29.0197 0 - 2.15363 2014-04-01 15:54:31,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 798 16 5816 5800 29.0685 22.6667 0.459202 2.15981 2014-04-01 15:54:32,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 799 16 5844 5828 29.1723 112 0.546279 2.15217 2014-04-01 15:54:33,501.501 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:54:33.417836min lat: 0.063861 max lat: 135.172 avg lat: 2.14946 2014-04-01 15:54:33,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:54:33,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 800 16 5855 5839 29.1908 44 1.20384 2.14946 2014-04-01 15:54:33,984.984 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:34,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 801 16 5877 5861 29.2642 88 0.439462 2.14492 2014-04-01 15:54:34,665.665 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:35,437.437 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:35,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 802 16 5881 5865 29.2477 16 1.05624 2.14391 2014-04-01 15:54:36,225.225 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 994/30189 objects degraded (3.293%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7536369, u'degraded_objects': 994, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 42123397532, u'degraded_total': 30189, u'bytes_used': 36379705344, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 35797752, u'op_per_sec': 8, u'version': 439, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.293', u'bytes_avail': 2299133157376}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:36,225.225 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:36,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 803 16 5881 5865 29.2112 0 - 2.14391 2014-04-01 15:54:36,974.974 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:37,502.502 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 804 16 5881 5865 29.1749 0 - 2.14391 2014-04-01 15:54:38,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 805 16 5881 5865 29.1387 0 - 2.14391 2014-04-01 15:54:39,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 806 16 5881 5865 29.1025 0 - 2.14391 2014-04-01 15:54:40,506.506 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 807 16 5909 5893 29.2052 22.4 0.483994 2.14842 2014-04-01 15:54:40,585.585 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:41,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:41,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 808 16 5933 5917 29.2879 96 0.729052 2.14263 2014-04-01 15:54:42,072.072 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:42,503.503 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 809 16 5948 5932 29.3258 60 0.9295 2.13872 2014-04-01 15:54:42,853.853 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 988/30432 objects degraded (3.247%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2866160, u'degraded_objects': 988, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 42463136156, u'degraded_total': 30432, u'bytes_used': 36898828288, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 33438539, u'op_per_sec': 7, u'version': 443, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.247', u'bytes_avail': 2298614034432}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:42,854.854 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:43,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 810 16 5950 5934 29.2995 8 0.797616 2.13831 2014-04-01 15:54:43,714.714 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:44,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 811 16 5950 5934 29.2634 0 - 2.13831 2014-04-01 15:54:45,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 812 16 5950 5934 29.2273 0 - 2.13831 2014-04-01 15:54:47,141.141 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 813 16 5950 5934 29.1914 0 - 2.13831 2014-04-01 15:54:47,339.339 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:47,511.511 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 814 16 5969 5953 29.2489 19 0.185699 2.14497 2014-04-01 15:54:48,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:48,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 815 16 5987 5971 29.3013 72 1.07356 2.14086 2014-04-01 15:54:49,184.184 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:49,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 816 16 6015 5999 29.4026 112 0.354143 2.1332 2014-04-01 15:54:50,140.140 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 964/30549 objects degraded (3.156%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6440490, u'degraded_objects': 964, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 42626714012, u'degraded_total': 30549, u'bytes_used': 37359857664, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 20609570, u'op_per_sec': 4, u'version': 447, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.156', u'bytes_avail': 2298153005056}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:50,140.140 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:50,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 817 16 6028 6012 29.4303 52 1.2266 2.13062 2014-04-01 15:54:50,847.847 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:51,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 818 16 6032 6016 29.4139 16 1.58651 2.12975 2014-04-01 15:54:52,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 819 16 6032 6016 29.3779 0 - 2.12975 2014-04-01 15:54:53,504.504 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:54:53.420875min lat: 0.063861 max lat: 135.172 avg lat: 2.12975 2014-04-01 15:54:53,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:54:53,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 820 16 6032 6016 29.3421 0 - 2.12975 2014-04-01 15:54:54,475.475 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:54,506.506 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 821 16 6032 6016 29.3064 0 - 2.12975 2014-04-01 15:54:55,260.260 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:55,505.505 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 822 16 6032 6016 29.2707 0 - 2.12975 2014-04-01 15:54:55,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:54:56,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 823 16 6042 6026 29.2838 8 0.204937 2.13576 2014-04-01 15:54:56,966.966 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 964/30792 objects degraded (3.131%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 964, u'num_pgs': 138, u'data_bytes': 42966452636, u'degraded_total': 30792, u'bytes_used': 37683187712, u'op_per_sec': 1, u'write_bytes_sec': 6757874, u'version': 451, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.131', u'bytes_avail': 2297829675008}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:54:56,966.966 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:54:57,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 824 16 6066 6050 29.3647 96 0.548771 2.13506 2014-04-01 15:54:57,673.673 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:54:58,506.506 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 825 16 6083 6067 29.4115 68 0.385069 2.13087 2014-04-01 15:54:59,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 826 16 6087 6071 29.3953 16 0.248457 2.12985 2014-04-01 15:55:00,508.508 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 827 16 6088 6072 29.3646 4 0.896749 2.12965 2014-04-01 15:55:01,314.314 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:01,506.506 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 828 16 6088 6072 29.3291 0 - 2.12965 2014-04-01 15:55:02,085.085 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:02,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 829 16 6088 6072 29.2937 0 - 2.12965 2014-04-01 15:55:02,872.872 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:03,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 830 16 6088 6072 29.2584 0 - 2.12965 2014-04-01 15:55:03,591.591 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 948/30966 objects degraded (3.061%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2838804, u'degraded_objects': 948, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 43209722268, u'degraded_total': 30966, u'bytes_used': 38220951552, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 49205940, u'op_per_sec': 11, u'version': 454, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.061', u'bytes_avail': 2297291911168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:03,592.592 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:04,296.296 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:04,508.508 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 831 16 6110 6094 29.3291 22 0.553625 2.13685 2014-04-01 15:55:05,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 832 16 6138 6122 29.4285 112 0.396978 2.12964 2014-04-01 15:55:06,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 833 16 6149 6133 29.4459 44 1.40353 2.12749 2014-04-01 15:55:07,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 834 16 6160 6144 29.4634 44 0.488197 2.12552 2014-04-01 15:55:08,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:08,508.508 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 835 16 6161 6145 29.4329 4 0.427925 2.12525 2014-04-01 15:55:09,277.277 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:09,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 836 16 6161 6145 29.3977 0 - 2.12525 2014-04-01 15:55:09,916.916 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:10,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 837 16 6161 6145 29.3626 0 - 2.12525 2014-04-01 15:55:10,698.698 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 946/31143 objects degraded (3.038%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9874599, u'degraded_objects': 946, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 43457186204, u'degraded_total': 31143, u'bytes_used': 38630141952, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 55544621, u'op_per_sec': 13, u'version': 458, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.038', u'bytes_avail': 2296882720768}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:10,699.699 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:11,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 838 16 6161 6145 29.3275 0 - 2.12525 2014-04-01 15:55:11,536.536 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:12,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 839 16 6161 6145 29.2926 0 - 2.12525 2014-04-01 15:55:13,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:55:13.423933min lat: 0.063861 max lat: 135.172 avg lat: 2.13018 2014-04-01 15:55:13,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:55:13,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 840 16 6193 6177 29.41 25.6 0.594738 2.13018 2014-04-01 15:55:14,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 841 16 6212 6196 29.4654 76 0.690923 2.12515 2014-04-01 15:55:15,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:15,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 842 16 6220 6204 29.4684 32 1.40477 2.1236 2014-04-01 15:55:15,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:16,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 843 16 6220 6204 29.4335 0 - 2.1236 2014-04-01 15:55:16,874.874 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:17,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 844 16 6220 6204 29.3986 0 - 2.1236 2014-04-01 15:55:17,703.703 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 936/31281 objects degraded (2.992%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 936, u'num_pgs': 138, u'data_bytes': 43650124188, u'degraded_total': 31281, u'bytes_used': 38968332288, u'op_per_sec': 7, u'write_bytes_sec': 32268638, u'version': 461, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.992', u'bytes_avail': 2296544530432}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:17,704.704 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:18,446.446 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:18,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 845 16 6228 6212 29.4017 10.6667 3.79304 2.12623 2014-04-01 15:55:19,511.511 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 846 16 6249 6233 29.4662 84 0.362083 2.12477 2014-04-01 15:55:20,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 847 16 6273 6257 29.5447 96 0.797866 2.11935 2014-04-01 15:55:21,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 848 16 6283 6267 29.5571 40 0.266333 2.11727 2014-04-01 15:55:22,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:22,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 849 16 6288 6272 29.5458 20 0.15589 2.11678 2014-04-01 15:55:23,005.005 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:23,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 850 16 6291 6275 29.5252 12 0.240296 2.11623 2014-04-01 15:55:23,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:24,506.506 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 938/31527 objects degraded (2.975%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6222892, u'degraded_objects': 938, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 43994057116, u'degraded_total': 31527, u'bytes_used': 39569559552, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 63473500, u'op_per_sec': 15, u'version': 465, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.975', u'bytes_avail': 2295943303168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:24,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:24,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 851 16 6291 6275 29.4905 0 - 2.11623 2014-04-01 15:55:25,257.257 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:25,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 852 16 6291 6275 29.4558 0 - 2.11623 2014-04-01 15:55:26,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 853 16 6306 6290 29.4916 20 0.163848 2.12249 2014-04-01 15:55:27,511.511 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 854 16 6329 6313 29.5648 92 0.357293 2.11725 2014-04-01 15:55:28,511.511 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 855 16 6341 6325 29.5864 48 1.2087 2.11496 2014-04-01 15:55:28,871.871 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:29,440.440 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:29,762.762 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 856 16 6362 6346 29.6499 84 0.584863 2.11063 2014-04-01 15:55:30,372.372 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:30,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 857 16 6365 6349 29.6293 12 0.936022 2.10998 2014-04-01 15:55:31,145.145 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:54:33.206665', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438604212, u'health': u'HEALTH_OK', u'kb_used': 9724772, u'store_stats': {u'bytes_total': 6317673, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:54:33.949425', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438360544, u'health': u'HEALTH_OK', u'kb_used': 9968440, u'store_stats': {u'bytes_total': 8415471, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:54:33.490945', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438603740, u'health': u'HEALTH_OK', u'kb_used': 9725244, u'store_stats': {u'bytes_total': 8414769, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 914/31635 objects degraded (2.889%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8386051, u'degraded_objects': 914, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 44145052060, u'degraded_total': 31635, u'bytes_used': 39842709504, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 27953503, u'op_per_sec': 6, u'version': 468, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.889', u'bytes_avail': 2295670153216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:31,146.146 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:31,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 858 16 6365 6349 29.5948 0 - 2.10998 2014-04-01 15:55:31,959.959 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:32,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 859 16 6365 6349 29.5603 0 - 2.10998 2014-04-01 15:55:33,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:55:33.427002min lat: 0.063861 max lat: 135.172 avg lat: 2.10998 2014-04-01 15:55:33,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:55:33,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 860 16 6365 6349 29.526 0 - 2.10998 2014-04-01 15:55:34,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 861 16 6379 6363 29.5567 14 4.72821 2.11501 2014-04-01 15:55:35,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 862 16 6400 6384 29.6199 84 0.379083 2.11244 2014-04-01 15:55:35,731.731 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:36,557.557 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:36,763.763 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 863 15 6415 6400 29.6597 64 1.61463 2.10932 2014-04-01 15:55:37,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 864 16 6428 6412 29.6809 48 0.839536 2.10717 2014-04-01 15:55:37,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:38,395.395 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.023529', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.071493', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 929/31914 objects degraded (2.911%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 921117, u'degraded_objects': 929, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 44535122332, u'degraded_total': 31914, u'bytes_used': 40348532736, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 39608044, u'op_per_sec': 9, u'version': 472, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.911', u'bytes_avail': 2295164329984}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:38,395.395 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:38,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 865 16 6437 6421 29.6882 36 0.121768 2.10515 2014-04-01 15:55:39,220.220 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:39,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 866 16 6437 6421 29.6539 0 - 2.10515 2014-04-01 15:55:40,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 867 16 6437 6421 29.6197 0 - 2.10515 2014-04-01 15:55:41,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 868 16 6437 6421 29.5856 0 - 2.10515 2014-04-01 15:55:42,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 869 16 6457 6441 29.6436 20 0.568437 2.10986 2014-04-01 15:55:42,945.945 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:43,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 870 16 6471 6455 29.6739 56 0.800364 2.1074 2014-04-01 15:55:43,779.779 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:44,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 871 16 6486 6470 29.7087 60 0.364316 2.10452 2014-04-01 15:55:44,560.560 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:45,486.486 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 913/32046 objects degraded (2.849%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7408886, u'degraded_objects': 913, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 44719671708, u'degraded_total': 32046, u'bytes_used': 40856051712, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 32105174, u'op_per_sec': 7, u'version': 476, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.849', u'bytes_avail': 2294656811008}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:45,486.486 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:45,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 872 16 6492 6476 29.7021 24 0.932955 2.10356 2014-04-01 15:55:46,229.229 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:46,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 873 16 6492 6476 29.6681 0 - 2.10356 2014-04-01 15:55:47,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 874 16 6492 6476 29.6342 0 - 2.10356 2014-04-01 15:55:48,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 875 16 6492 6476 29.6003 0 - 2.10356 2014-04-01 15:55:49,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 876 16 6492 6476 29.5665 0 - 2.10356 2014-04-01 15:55:50,147.147 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:50,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 877 16 6516 6500 29.6422 19.2 0.598664 2.10993 2014-04-01 15:55:50,925.925 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:51,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 878 16 6530 6514 29.6723 56 0.914972 2.10696 2014-04-01 15:55:51,563.563 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:52,290.290 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 916/32184 objects degraded (2.846%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 916, u'num_pgs': 138, u'data_bytes': 44912609692, u'degraded_total': 32184, u'bytes_used': 41227862016, u'op_per_sec': 2, u'write_bytes_sec': 12275733, u'version': 480, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.846', u'bytes_avail': 2294285000704}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:52,290.290 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:52,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 879 16 6535 6519 29.6612 20 0.132628 2.10567 2014-04-01 15:55:53,071.071 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:53,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:55:53.429961min lat: 0.063861 max lat: 135.172 avg lat: 2.10587 2014-04-01 15:55:53,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:55:53,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 880 16 6543 6527 29.6639 32 2.77254 2.10587 2014-04-01 15:55:54,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 881 16 6544 6528 29.6348 4 2.57782 2.10594 2014-04-01 15:55:55,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 882 16 6544 6528 29.6012 0 - 2.10594 2014-04-01 15:55:56,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 883 16 6544 6528 29.5676 0 - 2.10594 2014-04-01 15:55:56,691.691 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:57,511.511 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:55:57,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 884 16 6549 6533 29.5568 6.66667 5.83777 2.10899 2014-04-01 15:55:58,305.305 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:55:58,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 885 16 6575 6559 29.6409 104 0.786397 2.10901 2014-04-01 15:55:59,386.386 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 909/32340 objects degraded (2.811%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6665059, u'degraded_objects': 909, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 45130713500, u'degraded_total': 32340, u'bytes_used': 41625927680, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 24755936, u'op_per_sec': 5, u'version': 483, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.811', u'bytes_avail': 2293886935040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:55:59,386.386 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:55:59,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 886 16 6591 6575 29.6797 64 0.859681 2.10577 2014-04-01 15:56:00,229.229 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:00,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 887 16 6604 6588 29.7048 52 0.238413 2.1031 2014-04-01 15:56:01,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 888 16 6614 6598 29.7164 40 0.542929 2.1016 2014-04-01 15:56:02,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 889 16 6614 6598 29.683 0 - 2.1016 2014-04-01 15:56:03,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 890 16 6614 6598 29.6496 0 - 2.1016 2014-04-01 15:56:03,998.998 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:04,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 891 16 6614 6598 29.6164 0 - 2.1016 2014-04-01 15:56:05,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 892 16 6623 6607 29.6235 9 5.30643 2.10585 2014-04-01 15:56:06,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 893 16 6649 6633 29.7068 104 0.529113 2.10337 2014-04-01 15:56:07,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 894 16 6654 6638 29.6959 20 1.48006 2.10242 2014-04-01 15:56:07,704.704 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:08,476.476 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:08,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 895 16 6665 6649 29.7119 44 1.78044 2.1019 2014-04-01 15:56:09,284.284 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 921/32610 objects degraded (2.824%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8301858, u'degraded_objects': 921, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 45508200860, u'degraded_total': 32610, u'bytes_used': 42158219264, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 34393413, u'op_per_sec': 8, u'version': 489, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.824', u'bytes_avail': 2293354643456}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:09,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:09,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 896 16 6670 6654 29.7011 20 0.180165 2.10121 2014-04-01 15:56:09,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:10,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 897 16 6670 6654 29.668 0 - 2.10121 2014-04-01 15:56:11,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 898 16 6670 6654 29.6349 0 - 2.10121 2014-04-01 15:56:12,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 899 16 6670 6654 29.6019 0 - 2.10121 2014-04-01 15:56:13,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:56:13.432951min lat: 0.063861 max lat: 135.172 avg lat: 2.10399 2014-04-01 15:56:13,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:56:13,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 900 16 6697 6681 29.689 27 0.512354 2.10399 2014-04-01 15:56:13,594.594 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:14,170.170 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:14,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 901 16 6714 6698 29.7316 68 0.266847 2.10047 2014-04-01 15:56:15,098.098 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:15,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 902 16 6731 6715 29.774 68 1.40272 2.09727 2014-04-01 15:56:15,878.878 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 902/32733 objects degraded (2.756%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5825106, u'degraded_objects': 902, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 45680167324, u'degraded_total': 32733, u'bytes_used': 42550988800, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 26212980, u'op_per_sec': 6, u'version': 492, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.756', u'bytes_avail': 2292961873920}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:15,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:16,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 903 16 6738 6722 29.772 28 0.919631 2.09651 2014-04-01 15:56:16,630.630 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:17,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 904 16 6740 6724 29.7479 8 0.31099 2.09596 2014-04-01 15:56:18,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 905 16 6741 6725 29.7195 4 1.25382 2.09583 2014-04-01 15:56:19,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 906 16 6741 6725 29.6867 0 - 2.09583 2014-04-01 15:56:20,238.238 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:20,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 907 16 6757 6741 29.7245 32 2.47701 2.10093 2014-04-01 15:56:20,970.970 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:21,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 908 16 6775 6759 29.771 72 0.391837 2.09676 2014-04-01 15:56:21,770.770 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:22,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 909 16 6803 6787 29.8615 112 0.375059 2.09117 2014-04-01 15:56:22,743.743 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 905/32994 objects degraded (2.743%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7430681, u'degraded_objects': 905, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 46045071772, u'degraded_total': 32994, u'bytes_used': 43053879296, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 35295737, u'op_per_sec': 8, u'version': 496, u'pgs_by_state': [{u'count': 132, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.743', u'bytes_avail': 2292458983424}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:22,743.743 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:23,521.521 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:23,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 910 16 6811 6795 29.8638 32 1.08443 2.14075 2014-04-01 15:56:24,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 911 16 6814 6798 29.8442 12 0.155941 2.14011 2014-04-01 15:56:25,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 912 16 6814 6798 29.8115 0 - 2.14011 2014-04-01 15:56:26,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 913 16 6814 6798 29.7788 0 - 2.14011 2014-04-01 15:56:27,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:27,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 914 16 6814 6798 29.7462 0 - 2.14011 2014-04-01 15:56:28,467.467 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:28,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 915 16 6836 6820 29.8099 22 0.673193 2.14628 2014-04-01 15:56:29,322.322 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:29,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 916 16 6859 6843 29.8778 92 0.538656 2.14139 2014-04-01 15:56:30,119.119 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:55:33.206960', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554464, u'health': u'HEALTH_OK', u'kb_used': 9774520, u'store_stats': {u'bytes_total': 7366249, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123353}}, {u'last_updated': u'2014-04-01 06:55:33.959001', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438281292, u'health': u'HEALTH_OK', u'kb_used': 10047692, u'store_stats': {u'bytes_total': 10512623, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123999}}, {u'last_updated': u'2014-04-01 06:55:33.491220', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438554448, u'health': u'HEALTH_OK', u'kb_used': 9774536, u'store_stats': {u'bytes_total': 10511921, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2123297}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 840/33144 objects degraded (2.534%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3857326, u'degraded_objects': 840, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 46254786972, u'degraded_total': 33144, u'bytes_used': 43469647872, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 33430164, u'op_per_sec': 7, u'version': 499, u'pgs_by_state': [{u'count': 133, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.534', u'bytes_avail': 2292043214848}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:30,120.120 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:30,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 917 16 6878 6862 29.9281 76 0.425495 2.13715 2014-04-01 15:56:30,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:31,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 918 16 6880 6864 29.9042 8 1.47623 2.1369 2014-04-01 15:56:32,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 919 16 6881 6865 29.876 4 0.242848 2.13662 2014-04-01 15:56:33,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:56:33.436061min lat: 0.063861 max lat: 348.961 avg lat: 2.13662 2014-04-01 15:56:33,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:56:33,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 920 16 6881 6865 29.8435 0 - 2.13662 2014-04-01 15:56:34,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 921 16 6881 6865 29.8111 0 - 2.13662 2014-04-01 15:56:34,679.679 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:35,390.390 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:35,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 922 16 6881 6865 29.7788 0 - 2.13662 2014-04-01 15:56:36,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:36,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 923 16 6905 6889 29.8505 24 0.616138 2.14316 2014-04-01 15:56:37,130.130 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 847/33360 objects degraded (2.539%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 847, u'num_pgs': 138, u'data_bytes': 46556776860, u'degraded_total': 33360, u'bytes_used': 43692470272, u'op_per_sec': 2, u'write_bytes_sec': 10682324, u'version': 503, u'pgs_by_state': [{u'count': 133, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.539', u'bytes_avail': 2291820392448}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:37,131.131 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:37,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 924 16 6928 6912 29.9177 92 0.815394 2.13774 2014-04-01 15:56:38,189.189 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:38,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 925 16 6936 6920 29.92 32 0.28989 2.13606 2014-04-01 15:56:39,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 926 16 6957 6941 29.9784 84 0.650551 2.13359 2014-04-01 15:56:40,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 927 16 6960 6944 29.959 12 0.273816 2.13298 2014-04-01 15:56:41,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 928 16 6960 6944 29.9267 0 - 2.13298 2014-04-01 15:56:41,978.978 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:42,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 929 16 6960 6944 29.8945 0 - 2.13298 2014-04-01 15:56:42,746.746 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:43,372.372 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:43,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 930 16 6960 6944 29.8623 0 - 2.13298 2014-04-01 15:56:44,135.135 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 680/33558 objects degraded (2.026%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6627201, u'degraded_objects': 680, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 46833600924, u'degraded_total': 33558, u'bytes_used': 44220301312, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 45443670, u'op_per_sec': 10, u'version': 506, u'pgs_by_state': [{u'count': 134, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.026', u'bytes_avail': 2291292561408}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:44,135.135 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:44,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 931 16 6960 6944 29.8303 0 - 2.13298 2014-04-01 15:56:45,157.157 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:45,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 932 16 6988 6972 29.9184 22.4 0.442405 2.13833 2014-04-01 15:56:46,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 933 16 7010 6994 29.9807 88 1.273 2.13354 2014-04-01 15:56:47,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 934 16 7029 7013 30.0299 76 1.09271 2.13025 2014-04-01 15:56:48,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 935 16 7034 7018 30.0192 20 0.208511 2.12937 2014-04-01 15:56:48,902.902 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:49,669.669 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:49,823.823 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 936 16 7036 7020 29.9957 8 0.770319 2.12907 2014-04-01 15:56:50,448.448 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:50,523.523 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 937 16 7036 7020 29.9637 0 - 2.12907 2014-04-01 15:56:51,172.172 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 684/33729 objects degraded (2.028%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10091692, u'degraded_objects': 684, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 47072676252, u'degraded_total': 33729, u'bytes_used': 44567314432, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 74005741, u'op_per_sec': 17, u'version': 510, u'pgs_by_state': [{u'count': 134, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.028', u'bytes_avail': 2290945548288}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:51,173.173 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:51,523.523 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 938 16 7036 7020 29.9317 0 - 2.12907 2014-04-01 15:56:51,985.985 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:52,523.523 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 939 16 7036 7020 29.8998 0 - 2.12907 2014-04-01 15:56:53,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:56:53.438784min lat: 0.063861 max lat: 348.961 avg lat: 2.13632 2014-04-01 15:56:53,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:56:53,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 940 16 7051 7035 29.9318 15 5.49596 2.13632 2014-04-01 15:56:54,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 941 16 7074 7058 29.9978 92 0.360778 2.13251 2014-04-01 15:56:55,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 942 16 7093 7077 30.0466 76 0.487272 2.12926 2014-04-01 15:56:55,576.576 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:56:55.490923 7f829a7f3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a000be20 sd=14 :33605 s=2 pgs=12 cs=1 l=1 c=0x7f82a000c080).injecting socket failure 2014-04-01 15:56:55,599.599 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:56,165.165 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:56,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 943 16 7119 7103 30.125 104 0.356862 2.12364 2014-04-01 15:56:57,119.119 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:56:57,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 944 16 7125 7109 30.1185 24 0.772671 2.12246 2014-04-01 15:56:58,023.023 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 587/33900 objects degraded (1.732%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 587, u'num_pgs': 138, u'data_bytes': 47311751580, u'degraded_total': 33900, u'bytes_used': 44990541824, u'op_per_sec': 9, u'write_bytes_sec': 38475891, u'version': 514, u'pgs_by_state': [{u'count': 135, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.732', u'bytes_avail': 2290522320896}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:56:58,024.024 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:56:58,523.523 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 945 16 7125 7109 30.0867 0 - 2.12246 2014-04-01 15:56:58,722.722 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:56:59,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 946 16 7125 7109 30.0548 0 - 2.12246 2014-04-01 15:57:00,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 947 16 7125 7109 30.0231 0 - 2.12246 2014-04-01 15:57:01,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 948 16 7143 7127 30.0674 18 4.14241 2.12789 2014-04-01 15:57:02,330.330 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:02,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 949 16 7158 7142 30.0989 60 1.20388 2.12515 2014-04-01 15:57:03,050.050 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:03,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 950 16 7186 7170 30.185 112 0.450892 2.11965 2014-04-01 15:57:03,847.847 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:04,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 951 16 7208 7192 30.2458 88 0.983379 2.1156 2014-04-01 15:57:04,915.915 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 575/34146 objects degraded (1.684%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 575, u'num_pgs': 138, u'data_bytes': 47655684508, u'degraded_total': 34146, u'bytes_used': 45531967488, u'op_per_sec': 8, u'write_bytes_sec': 34396041, u'version': 519, u'pgs_by_state': [{u'count': 135, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.684', u'bytes_avail': 2289980895232}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:04,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:05,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 952 16 7218 7202 30.256 40 0.297374 2.11337 2014-04-01 15:57:05,665.665 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:06,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 953 16 7219 7203 30.2285 4 1.02322 2.11322 2014-04-01 15:57:07,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 954 16 7236 7220 30.2681 68 0.379349 2.11296 2014-04-01 15:57:08,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 955 16 7238 7222 30.2448 8 0.366369 2.11245 2014-04-01 15:57:09,329.329 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:09,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 956 16 7238 7222 30.2131 0 - 2.11245 2014-04-01 15:57:10,100.100 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:10,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 957 16 7246 7230 30.215 16 3.50278 2.11435 2014-04-01 15:57:10,897.897 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:11,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 958 16 7261 7245 30.2461 60 0.853066 2.11443 2014-04-01 15:57:11,648.648 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 581/34365 objects degraded (1.691%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3727967, u'degraded_objects': 581, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 47961868700, u'degraded_total': 34365, u'bytes_used': 45891543040, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 65860760, u'op_per_sec': 15, u'version': 522, u'pgs_by_state': [{u'count': 135, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.691', u'bytes_avail': 2289621319680}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:11,649.649 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:12,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 959 16 7282 7266 30.3021 84 0.713646 2.11072 2014-04-01 15:57:12,821.821 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:13,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:57:13.444419min lat: 0.063861 max lat: 348.961 avg lat: 2.1054 2014-04-01 15:57:13,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:57:13,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 960 16 7310 7294 30.3872 112 0.525899 2.1054 2014-04-01 15:57:14,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 961 16 7325 7309 30.418 60 0.850576 2.10264 2014-04-01 15:57:15,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 962 16 7338 7322 30.4404 52 0.524258 2.10052 2014-04-01 15:57:16,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 963 16 7340 7324 30.4171 8 0.398527 2.10009 2014-04-01 15:57:16,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:17,518.518 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:17,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 964 16 7340 7324 30.3856 0 - 2.10009 2014-04-01 15:57:18,167.167 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:18,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 965 16 7340 7324 30.3541 0 - 2.10009 2014-04-01 15:57:18,937.937 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 575/34716 objects degraded (1.656%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9900136, u'degraded_objects': 575, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 48452602268, u'degraded_total': 34716, u'bytes_used': 46750060544, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 76726054, u'op_per_sec': 18, u'version': 527, u'pgs_by_state': [{u'count': 135, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.656', u'bytes_avail': 2288762802176}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:18,938.938 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:19,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 966 16 7340 7324 30.3226 0 - 2.10009 2014-04-01 15:57:19,900.900 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:20,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 967 16 7343 7327 30.3037 3 6.65256 2.10194 2014-04-01 15:57:21,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 968 16 7376 7360 30.4087 132 0.306053 2.10418 2014-04-01 15:57:22,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 969 16 7393 7377 30.4475 68 0.563471 2.1005 2014-04-01 15:57:23,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:23,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 970 16 7406 7390 30.4697 52 0.36347 2.09859 2014-04-01 15:57:24,227.227 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:24,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 971 16 7410 7394 30.4548 16 0.299491 2.09779 2014-04-01 15:57:25,010.010 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:25,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 972 16 7410 7394 30.4235 0 - 2.09779 2014-04-01 15:57:25,707.707 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 566/34866 objects degraded (1.623%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 22261723, u'degraded_objects': 566, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 48662317468, u'degraded_total': 34866, u'bytes_used': 47042256896, u'recovering_objects_per_sec': 5, u'write_bytes_sec': 65072731, u'op_per_sec': 15, u'version': 531, u'pgs_by_state': [{u'count': 135, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.623', u'bytes_avail': 2288470605824}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:25,708.708 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:26,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:26,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 973 16 7410 7394 30.3922 0 - 2.09779 2014-04-01 15:57:27,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 974 16 7410 7394 30.361 0 - 2.09779 2014-04-01 15:57:28,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 975 16 7410 7394 30.3299 0 - 2.09779 2014-04-01 15:57:29,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 976 16 7430 7414 30.3808 16 0.252658 2.10602 2014-04-01 15:57:30,082.082 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:30,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 977 16 7453 7437 30.4438 92 0.611451 2.10168 2014-04-01 15:57:30,835.835 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:31,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 978 16 7474 7458 30.4986 84 0.815262 2.09752 2014-04-01 15:57:31,632.632 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:32,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 979 16 7496 7480 30.5573 88 0.633198 2.09365 2014-04-01 15:57:33,081.081 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:56:33.207236', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522852, u'health': u'HEALTH_OK', u'kb_used': 9806132, u'store_stats': {u'bytes_total': 3722579, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:56:33.959273', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438205468, u'health': u'HEALTH_OK', u'kb_used': 10123516, u'store_stats': {u'bytes_total': 4771371, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:56:33.491489', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438522692, u'health': u'HEALTH_OK', u'kb_used': 9806292, u'store_stats': {u'bytes_total': 4771160, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 405/35022 objects degraded (1.156%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 955705, u'degraded_objects': 405, u'num_pgs': 138, u'recovering_keys_per_sec': 0, u'data_bytes': 48880421276, u'degraded_total': 35022, u'bytes_used': 47429406720, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 37272508, u'op_per_sec': 8, u'version': 535, u'pgs_by_state': [{u'count': 136, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.156', u'bytes_avail': 2288083456000}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:33,081.081 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:33,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:57:33.447564min lat: 0.063861 max lat: 348.961 avg lat: 2.09249 2014-04-01 15:57:33,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:57:33,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 980 16 7503 7487 30.5547 28 0.344633 2.09249 2014-04-01 15:57:33,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:34,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 981 16 7503 7487 30.5235 0 - 2.09249 2014-04-01 15:57:35,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 982 16 7503 7487 30.4924 0 - 2.09249 2014-04-01 15:57:36,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 983 16 7503 7487 30.4614 0 - 2.09249 2014-04-01 15:57:37,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 984 16 7518 7502 30.4914 15 4.49173 2.09745 2014-04-01 15:57:37,952.952 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:38,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 985 16 7528 7512 30.5011 40 0.722234 2.0965 2014-04-01 15:57:38,640.640 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:39,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 986 16 7550 7534 30.5594 88 0.430927 2.09289 2014-04-01 15:57:39,770.770 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:39,948.948 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:57:39.837474 7f82a81ef700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6805/4943 pipe(0x7f82a0012140 sd=8 :40974 s=2 pgs=8 cs=1 l=1 c=0x7f82a00123a0).injecting socket failure 2014-04-01 15:57:40,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 987 16 7574 7558 30.6257 96 0.801476 2.08911 2014-04-01 15:57:40,670.670 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 398/35223 objects degraded (1.130%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 398, u'num_pgs': 138, u'data_bytes': 49161439644, u'degraded_total': 35223, u'bytes_used': 48088850432, u'op_per_sec': 6, u'write_bytes_sec': 27071235, u'version': 540, u'pgs_by_state': [{u'count': 136, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.130', u'bytes_avail': 2287424012288}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:40,670.670 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:41,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 988 16 7595 7579 30.6797 84 0.612155 2.08506 2014-04-01 15:57:41,626.626 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:42,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 989 16 7610 7594 30.7093 60 0.969625 2.08229 2014-04-01 15:57:43,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 990 16 7613 7597 30.6904 12 1.89906 2.08205 2014-04-01 15:57:44,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 991 16 7614 7598 30.6635 4 0.676321 2.08187 2014-04-01 15:57:45,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 992 16 7614 7598 30.6326 0 - 2.08187 2014-04-01 15:57:45,612.612 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:46,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:46,773.773 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 993 16 7614 7598 30.6017 0 - 2.08187 2014-04-01 15:57:47,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 994 16 7614 7598 30.5709 0 - 2.08187 2014-04-01 15:57:47,682.682 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:48,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 995 16 7638 7622 30.6367 24 0.585749 2.08836 2014-04-01 15:57:48,548.548 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 405/35553 objects degraded (1.139%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 405, u'num_pgs': 138, u'data_bytes': 49622813084, u'degraded_total': 35553, u'bytes_used': 48614723584, u'op_per_sec': 4, u'write_bytes_sec': 19565517, u'version': 544, u'pgs_by_state': [{u'count': 136, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.139', u'bytes_avail': 2286898139136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:48,549.549 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:49,274.274 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:49,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 996 16 7656 7640 30.6782 72 0.928697 2.0854 2014-04-01 15:57:50,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 997 15 7685 7670 30.7678 120 0.43714 2.07952 2014-04-01 15:57:51,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 998 16 7689 7673 30.749 12 1.02276 2.07906 2014-04-01 15:57:52,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 999 16 7690 7674 30.7222 4 1.33468 2.07896 2014-04-01 15:57:53,291.291 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:53,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:57:53.450722min lat: 0.063861 max lat: 348.961 avg lat: 2.07896 2014-04-01 15:57:53,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:57:53,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1000 16 7690 7674 30.6915 0 - 2.07896 2014-04-01 15:57:54,019.019 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:54,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1001 16 7690 7674 30.6608 0 - 2.07896 2014-04-01 15:57:54,711.711 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:57:55,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1002 16 7709 7693 30.706 25.3333 0.328075 2.08391 2014-04-01 15:57:55,543.543 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 404/35751 objects degraded (1.130%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 404, u'num_pgs': 138, u'data_bytes': 49899637148, u'degraded_total': 35751, u'bytes_used': 48984342528, u'op_per_sec': 9, u'write_bytes_sec': 39664716, u'version': 548, u'pgs_by_state': [{u'count': 136, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.130', u'bytes_avail': 2286528520192}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:57:55,544.544 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:57:56,288.288 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:57:56,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1003 16 7722 7706 30.7273 52 1.1873 2.08134 2014-04-01 15:57:57,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1004 16 7740 7724 30.7684 72 1.68946 2.07894 2014-04-01 15:57:58,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1005 16 7768 7752 30.8492 112 0.601261 2.07383 2014-04-01 15:57:59,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1006 16 7795 7779 30.9258 108 0.466617 2.06877 2014-04-01 15:58:00,056.056 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:00,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1007 16 7813 7797 30.9666 72 1.07787 2.06567 2014-04-01 15:58:00,683.683 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:01,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1008 16 7820 7804 30.9637 28 0.197503 2.06439 2014-04-01 15:58:01,782.782 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:58:02,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1009 16 7820 7804 30.933 0 - 2.06439 2014-04-01 15:58:02,565.565 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 400/35979 objects degraded (1.112%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 400, u'num_pgs': 138, u'data_bytes': 50218404252, u'degraded_total': 35979, u'bytes_used': 49397170176, u'op_per_sec': 13, u'write_bytes_sec': 58302558, u'version': 552, u'pgs_by_state': [{u'count': 136, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.112', u'bytes_avail': 2286115692544}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:58:02,566.566 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:03,537.537 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:03,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1010 16 7820 7804 30.9024 0 - 2.06439 2014-04-01 15:58:04,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1011 16 7820 7804 30.8718 0 - 2.06439 2014-04-01 15:58:05,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1012 16 7828 7812 30.8729 8 5.19567 2.0678 2014-04-01 15:58:06,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1013 16 7856 7840 30.953 112 0.236082 2.06712 2014-04-01 15:58:07,526.526 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:07,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1014 16 7874 7858 30.9934 72 1.05455 2.06405 2014-04-01 15:58:08,115.115 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:08,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1015 16 7906 7890 31.089 128 0.50859 2.05826 2014-04-01 15:58:08,931.931 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:58:09,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1016 16 7923 7907 31.1253 68 0.682202 2.05527 2014-04-01 15:58:09,821.821 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 224/36246 objects degraded (0.618%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 224, u'num_pgs': 138, u'data_bytes': 50591697308, u'degraded_total': 36246, u'bytes_used': 50100043776, u'op_per_sec': 7, u'write_bytes_sec': 30499246, u'version': 556, u'pgs_by_state': [{u'count': 137, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.618', u'bytes_avail': 2285412818944}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:58:09,822.822 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:10,516.516 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:10,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1017 16 7937 7921 31.1498 56 0.154495 2.05269 2014-04-01 15:58:11,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1018 16 7938 7922 31.1231 4 0.584825 2.05251 2014-04-01 15:58:12,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1019 16 7938 7922 31.0926 0 - 2.05251 2014-04-01 15:58:13,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:58:13.453727min lat: 0.063861 max lat: 348.961 avg lat: 2.05251 2014-04-01 15:58:13,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:58:13,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1020 16 7938 7922 31.0621 0 - 2.05251 2014-04-01 15:58:14,138.138 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:14,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1021 16 7954 7938 31.0943 21.3333 4.81229 2.05747 2014-04-01 15:58:14,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:15,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1022 16 7969 7953 31.1226 60 0.298257 2.05535 2014-04-01 15:58:16,000.000 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:58:16,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1023 16 7992 7976 31.1821 92 0.822427 2.0516 2014-04-01 15:58:16,809.809 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 214/36522 objects degraded (0.586%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 214, u'num_pgs': 138, u'data_bytes': 50977573276, u'degraded_total': 36522, u'bytes_used': 50765041664, u'op_per_sec': 15, u'write_bytes_sec': 64515800, u'version': 559, u'pgs_by_state': [{u'count': 137, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.586', u'bytes_avail': 2284747821056}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:58:16,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:17,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1024 16 8006 7990 31.2063 56 0.574103 2.04928 2014-04-01 15:58:17,675.675 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:18,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1025 16 8006 7990 31.1759 0 - 2.04928 2014-04-01 15:58:19,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1026 16 8006 7990 31.1455 0 - 2.04928 2014-04-01 15:58:20,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1027 16 8006 7990 31.1152 0 - 2.04928 2014-04-01 15:58:21,463.463 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:21,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1028 16 8006 7990 31.0849 0 - 2.04928 2014-04-01 15:58:22,409.409 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:22,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1029 16 8024 8008 31.1246 14.4 5.24381 2.05585 2014-04-01 15:58:23,381.381 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:58:23,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1030 16 8051 8035 31.1993 108 0.224747 2.05063 2014-04-01 15:58:24,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1031 16 8065 8049 31.2233 56 0.692872 2.04831 2014-04-01 15:58:24,616.616 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 217/36690 objects degraded (0.591%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 217, u'num_pgs': 138, u'data_bytes': 51212454300, u'degraded_total': 36690, u'bytes_used': 51121876992, u'op_per_sec': 0, u'write_bytes_sec': 2481921, u'version': 563, u'pgs_by_state': [{u'count': 137, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.591', u'bytes_avail': 2284390985728}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:58:24,616.616 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:25,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1032 16 8065 8049 31.1931 0 - 2.04831 2014-04-01 15:58:25,701.701 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:26,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1033 16 8065 8049 31.1629 0 - 2.04831 2014-04-01 15:58:27,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1034 16 8065 8049 31.1327 0 - 2.04831 2014-04-01 15:58:28,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1035 16 8083 8067 31.1722 18 4.00478 2.05218 2014-04-01 15:58:29,541.541 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:29,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1036 16 8109 8093 31.2425 104 0.702453 2.04788 2014-04-01 15:58:30,364.364 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:30,740.740 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1037 16 8123 8107 31.2664 56 0.879023 2.04592 2014-04-01 15:58:31,363.363 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:58:31,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1038 16 8152 8136 31.348 116 0.594636 2.04114 2014-04-01 15:58:32,292.292 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:57:33.207527', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474364, u'health': u'HEALTH_OK', u'kb_used': 9854620, u'store_stats': {u'bytes_total': 5819731, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:57:33.959541', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438108076, u'health': u'HEALTH_OK', u'kb_used': 10220908, u'store_stats': {u'bytes_total': 6868523, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:57:33.491732', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438474188, u'health': u'HEALTH_OK', u'kb_used': 9854796, u'store_stats': {u'bytes_total': 6868312, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 210/36999 objects degraded (0.568%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 210, u'num_pgs': 138, u'data_bytes': 51644467612, u'degraded_total': 36999, u'bytes_used': 51766784000, u'op_per_sec': 7, u'write_bytes_sec': 32646244, u'version': 567, u'pgs_by_state': [{u'count': 137, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.568', u'bytes_avail': 2283746078720}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:58:32,293.293 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:58:32,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1039 16 8168 8152 31.3794 64 0.753711 2.03832 2014-04-01 15:58:33,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:58:33.456585min lat: 0.063861 max lat: 348.961 avg lat: 2.03503 2014-04-01 15:58:33,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:58:33,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1040 16 8189 8173 31.43 84 0.676992 2.03503 2014-04-01 15:58:33,839.839 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:58:35,043.043 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1041 16 8189 8173 31.3998 0 - 2.03503 2014-04-01 15:58:35,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1042 16 8189 8173 31.3696 0 - 2.03503 2014-04-01 15:58:36,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1043 16 8189 8173 31.3396 0 - 2.03503 2014-04-01 15:59:11,272.272 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1044 16 8189 8173 31.3095 0 - 2.03503 2014-04-01 15:59:11,272.272 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1045 16 8199 8183 31.3179 8 5.58677 2.03953 2014-04-01 15:59:11,273.273 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1046 16 8223 8207 31.3797 96 0.916318 2.03839 2014-04-01 15:59:11,273.273 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1047 16 8246 8230 31.4376 92 0.544341 2.0351 2014-04-01 15:59:11,273.273 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1048 16 8269 8253 31.4953 92 0.559441 2.03136 2014-04-01 15:59:11,274.274 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1049 16 8293 8277 31.5568 96 0.601603 2.02728 2014-04-01 15:59:11,274.274 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1050 16 8320 8304 31.6296 108 0.303845 2.02297 2014-04-01 15:59:11,274.274 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1051 16 8338 8322 31.668 72 1.09876 2.02027 2014-04-01 15:59:11,275.275 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1052 16 8360 8344 31.7215 88 0.866765 2.01684 2014-04-01 15:59:11,275.275 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1053 16 8386 8370 31.7902 104 0.617645 2.01275 2014-04-01 15:59:11,275.275 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1054 16 8405 8389 31.8321 76 0.830276 2.00973 2014-04-01 15:59:11,276.276 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:58:43.189640 7f82986e5700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6805/4947 pipe(0x7f82a0019960 sd=12 :57425 s=2 pgs=10 cs=1 l=1 c=0x7f82a0019bc0).injecting socket failure 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1055 15 8433 8418 31.9119 116 0.7813 2.00514 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1056 16 8451 8435 31.946 68 0.658305 2.00238 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1057 16 8474 8458 32.0028 92 0.366947 1.9988 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1058 16 8503 8487 32.0822 116 0.645229 1.99422 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1059 16 8527 8511 32.1426 96 0.73206 1.99061 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:58:53.459355min lat: 0.063861 max lat: 348.961 avg lat: 1.98574 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:59:11,278.278 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1060 16 8556 8540 32.2217 116 0.450415 1.98574 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1061 16 8574 8558 32.2591 72 0.501962 1.98291 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1062 16 8590 8574 32.289 64 0.567981 1.98063 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1063 16 8590 8574 32.2586 0 - 1.98063 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1064 16 8590 8574 32.2283 0 - 1.98063 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1065 16 8590 8574 32.1981 0 - 1.98063 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1066 16 8603 8587 32.2166 13 0.139611 1.98298 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1067 16 8628 8612 32.2801 100 0.577254 1.98191 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1068 16 8654 8638 32.3473 104 0.531217 1.978 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1069 16 8677 8661 32.4031 92 0.896801 1.97458 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1070 16 8697 8681 32.4475 80 0.601192 1.97139 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1071 16 8725 8709 32.5218 112 0.757301 1.96727 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1072 16 8753 8737 32.5959 112 0.543417 1.96306 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1073 16 8777 8761 32.655 96 0.73375 1.95943 2014-04-01 15:59:11,279.279 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1074 16 8795 8779 32.6916 72 0.428705 1.95666 2014-04-01 15:59:11,280.280 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1075 16 8824 8808 32.7691 116 0.737511 1.95269 2014-04-01 15:59:11,280.280 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1076 16 8844 8828 32.813 80 1.06412 1.94961 2014-04-01 15:59:11,280.280 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1077 16 8865 8849 32.8605 84 0.721106 1.94656 2014-04-01 15:59:11,757.757 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1078 16 8892 8876 32.9302 108 0.893321 1.94289 2014-04-01 15:59:12,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1079 16 8916 8900 32.9886 96 0.517342 1.93916 2014-04-01 15:59:13,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:59:13.462200min lat: 0.063861 max lat: 348.961 avg lat: 1.93671 2014-04-01 15:59:13,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:59:13,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1080 16 8936 8920 33.0322 80 0.694592 1.93671 2014-04-01 15:59:14,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1081 16 8963 8947 33.1015 108 0.144606 1.93292 2014-04-01 15:59:15,059.059 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:59:15,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1082 16 8984 8968 33.1485 84 0.973949 1.93016 2014-04-01 15:59:15,932.932 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:59:16,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1083 15 9005 8990 33.1992 88 0.646187 1.9268 2014-04-01 15:59:16,727.727 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:59:17,532.532 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.019847', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.057298', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:58:33.238482', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438420732, u'health': u'HEALTH_OK', u'kb_used': 9908252, u'store_stats': {u'bytes_total': 6868307, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673987}}, {u'last_updated': u'2014-04-01 06:58:33.959844', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438019968, u'health': u'HEALTH_OK', u'kb_used': 10309016, u'store_stats': {u'bytes_total': 8965675, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2674203}}, {u'last_updated': u'2014-04-01 06:58:33.491998', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438420572, u'health': u'HEALTH_OK', u'kb_used': 9908412, u'store_stats': {u'bytes_total': 8965464, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2673992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 36 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'num_pgs': 138, u'data_bytes': 55020882332, u'bytes_used': 56477655040, u'op_per_sec': 33, u'version': 591, u'pgs_by_state': [{u'count': 138, u'state_name': u'active+clean'}], u'write_bytes_sec': 141728857, u'bytes_avail': 2279035207680}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 52, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 15:59:17,532.532 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 15:59:17,532.532 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:59:17,532.532 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 15:59:17,533.533 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:59:17,533.533 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:59:17,533.533 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 15:59:17,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1084 16 9039 9023 33.2903 132 0.559999 1.92197 2014-04-01 15:59:18,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1085 16 9066 9050 33.3591 108 0.478929 1.91817 2014-04-01 15:59:19,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1086 16 9081 9065 33.3837 60 0.388734 1.91579 2014-04-01 15:59:20,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1087 16 9095 9079 33.4045 56 0.438403 1.91431 2014-04-01 15:59:21,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1088 16 9124 9108 33.4804 116 0.657978 1.91091 2014-04-01 15:59:22,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1089 16 9134 9118 33.4863 40 0.805431 1.90961 2014-04-01 15:59:24,453.453 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1090 16 9134 9118 33.4556 0 - 1.90961 2014-04-01 15:59:24,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1091 16 9134 9118 33.4249 0 - 1.90961 2014-04-01 15:59:25,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1092 16 9134 9118 33.3943 0 - 1.90961 2014-04-01 15:59:26,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1093 16 9134 9118 33.3638 0 - 1.90961 2014-04-01 15:59:27,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1094 16 9160 9144 33.4283 20.8 0.361928 1.9143 2014-04-01 15:59:28,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1095 16 9172 9156 33.4416 48 1.12108 1.9129 2014-04-01 15:59:29,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1096 16 9189 9173 33.4732 68 0.411269 1.9108 2014-04-01 15:59:30,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1097 16 9214 9198 33.5338 100 0.610312 1.9079 2014-04-01 15:59:31,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1098 16 9246 9230 33.6198 128 0.505645 1.90304 2014-04-01 15:59:32,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1099 16 9250 9234 33.6038 16 0.553188 1.90249 2014-04-01 15:59:33,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:59:33.465181min lat: 0.063861 max lat: 348.961 avg lat: 1.90249 2014-04-01 15:59:33,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:59:33,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1100 16 9250 9234 33.5732 0 - 1.90249 2014-04-01 15:59:34,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1101 16 9250 9234 33.5427 0 - 1.90249 2014-04-01 15:59:34,877.877 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:59:34,877.877 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:59:34,877.877 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 15:59:34,878.878 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:59:35,716.716 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:59:37,993.993 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1102 16 9250 9234 33.5123 0 - 1.90249 2014-04-01 15:59:38,780.780 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1103 16 9264 9248 33.5327 14 4.71499 1.9069 2014-04-01 15:59:38,780.780 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1104 16 9279 9263 33.5566 60 0.811101 1.90601 2014-04-01 15:59:39,254.254 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1105 16 9285 9269 33.548 24 0.123238 1.90514 2014-04-01 15:59:39,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1106 16 9305 9289 33.59 80 0.341624 1.90471 2014-04-01 15:59:39,576.576 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 15:59:39,576.576 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 34' 2014-04-01 15:59:40,834.834 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1107 16 9330 9314 33.65 100 0.635513 1.90146 2014-04-01 15:59:40,840.840 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 34 2014-04-01 15:59:41,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1108 15 9354 9339 33.7098 100 0.53916 1.89807 2014-04-01 15:59:42,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1109 15 9383 9368 33.784 116 0.412793 1.89404 2014-04-01 15:59:43,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1110 16 9393 9377 33.786 36 0.439162 1.89271 2014-04-01 15:59:44,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1111 16 9402 9386 33.788 36 0.888443 1.89219 2014-04-01 15:59:45,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1112 16 9412 9396 33.7936 40 1.04 1.89206 2014-04-01 15:59:45,850.850 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:59:45,850.850 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:59:45,850.850 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 15:59:45,850.850 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:59:46,635.635 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:59:47,358.358 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1113 16 9412 9396 33.7632 0 - 1.89206 2014-04-01 15:59:48,421.421 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1114 16 9412 9396 33.7329 0 - 1.89206 2014-04-01 15:59:48,982.982 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1115 16 9412 9396 33.7026 0 - 1.89206 2014-04-01 15:59:49,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1116 16 9431 9415 33.7405 19 0.253551 1.89584 2014-04-01 15:59:50,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1117 16 9452 9436 33.7855 84 0.860598 1.89328 2014-04-01 15:59:51,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1118 16 9476 9460 33.8412 96 0.742761 1.89036 2014-04-01 15:59:52,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1119 16 9500 9484 33.8967 96 0.707782 1.88729 2014-04-01 15:59:53,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:59:53.468145min lat: 0.063861 max lat: 348.961 avg lat: 1.88436 2014-04-01 15:59:53,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:59:53,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1120 16 9524 9508 33.9521 96 0.545685 1.88436 2014-04-01 15:59:54,028.028 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:59:54,028.028 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:59:54,028.028 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 15:59:54,028.028 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 15:59:54,028.028 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 15:59:54,029.029 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 15:59:54,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1121 16 9548 9532 34.0075 96 0.593741 1.88134 2014-04-01 15:59:55,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1122 16 9579 9563 34.0877 124 0.352816 1.87708 2014-04-01 15:59:56,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1123 16 9593 9577 34.1072 56 1.24934 1.87576 2014-04-01 15:59:57,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1124 16 9611 9595 34.1409 72 0.637332 1.87359 2014-04-01 15:59:58,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1125 16 9618 9602 34.1354 28 1.79075 1.87322 2014-04-01 15:59:59,535.535 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 15:59:59,536.536 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 15:59:59,536.536 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool data 2014-04-01 15:59:59,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:59:59,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1126 16 9628 9612 34.1406 40 1.17049 1.8729 2014-04-01 16:00:00,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:00:00,717.717 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1127 16 9628 9612 34.1103 0 - 1.8729 2014-04-01 16:00:01,193.193 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 34' 2014-04-01 16:00:01,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1128 16 9628 9612 34.0801 0 - 1.8729 2014-04-01 16:00:02,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1129 16 9628 9612 34.0499 0 - 1.8729 2014-04-01 16:00:02,719.719 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pgp_num to 34 2014-04-01 16:00:03,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1130 16 9637 9621 34.0516 9 0.553657 1.87449 2014-04-01 16:00:04,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1131 16 9647 9631 34.0569 40 5.95295 1.87823 2014-04-01 16:00:05,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1132 16 9670 9654 34.108 92 0.769763 1.87554 2014-04-01 16:00:06,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1133 16 9700 9684 34.1838 120 0.557581 1.87181 2014-04-01 16:00:07,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1134 16 9716 9700 34.2101 64 1.43338 1.87011 2014-04-01 16:00:07,736.736 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:07,736.736 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:07,737.737 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 16:00:07,737.737 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:00:07,737.737 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:00:07,738.738 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 16:00:08,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1135 16 9723 9707 34.2046 28 0.271158 1.86915 2014-04-01 16:00:09,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1136 16 9723 9707 34.1745 0 - 1.86915 2014-04-01 16:00:10,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1137 16 9723 9707 34.1445 0 - 1.86915 2014-04-01 16:00:11,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1138 16 9723 9707 34.1145 0 - 1.86915 2014-04-01 16:00:12,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1139 16 9743 9727 34.1547 20 0.831899 1.87282 2014-04-01 16:00:13,410.410 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3] out_osds: [2] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:13,410.410 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:13,411.411 INFO:teuthology.task.thrashosds.thrasher:Adding osd 2 2014-04-01 16:00:13,411.411 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-04-01 16:00:13,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:00:13.471214min lat: 0.063861 max lat: 348.961 avg lat: 1.86922 2014-04-01 16:00:13,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:00:13,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1140 16 9772 9756 34.2265 116 0.486938 1.86922 2014-04-01 16:00:14,262.262 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.2. 2014-04-01 16:00:14,273.273 INFO:teuthology.task.thrashosds.thrasher:Added osd 2 2014-04-01 16:00:14,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1141 16 9791 9775 34.2631 76 0.462134 1.86701 2014-04-01 16:00:15,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1142 16 9796 9780 34.2506 20 0.303981 1.8666 2014-04-01 16:00:16,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1143 16 9803 9787 34.2452 28 0.144043 1.86592 2014-04-01 16:00:17,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1144 16 9804 9788 34.2187 4 1.09364 1.86584 2014-04-01 16:00:18,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1145 16 9804 9788 34.1888 0 - 1.86584 2014-04-01 16:00:19,273.273 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:19,274.274 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:19,275.275 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 16:00:19,275.275 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:00:19,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1146 16 9808 9792 34.173 8 5.11417 1.86689 2014-04-01 16:00:20,027.027 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:00:20,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1147 16 9829 9813 34.2164 84 0.632906 1.86678 2014-04-01 16:00:20,982.982 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 16:00:20,982.982 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pg_num 54' 2014-04-01 16:00:21,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1148 16 9847 9831 34.2493 72 0.697201 1.86413 2014-04-01 16:00:22,036.036 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pg_num to 54 2014-04-01 16:00:22,430.430 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:00:22.344226 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a000dbe0 sd=11 :57618 s=2 pgs=3 cs=1 l=1 c=0x7f82a0013820).injecting socket failure 2014-04-01 16:00:22,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1149 16 9857 9841 34.2543 40 0.918733 1.86324 2014-04-01 16:00:23,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1150 16 9869 9853 34.2663 48 0.22425 1.86159 2014-04-01 16:00:24,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1151 16 9877 9861 34.2643 32 0.134851 1.86024 2014-04-01 16:00:25,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1152 16 9877 9861 34.2345 0 - 1.86024 2014-04-01 16:00:26,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1153 16 9880 9864 34.2153 6 4.58029 1.86124 2014-04-01 16:00:27,048.048 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:27,048.048 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:27,049.049 INFO:teuthology.task.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.228218 2014-04-01 16:00:27,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.228218439512' 2014-04-01 16:00:27,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1154 16 9891 9875 34.2237 44 0.463565 1.86082 2014-04-01 16:00:28,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1155 16 9909 9893 34.2564 72 0.372475 1.85998 2014-04-01 16:00:28,880.880 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.0 primary-affinity to 0.228218 (8149562) 2014-04-01 16:00:29,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1156 16 9912 9896 34.2372 12 0.141257 1.85955 2014-04-01 16:00:30,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1157 16 9919 9903 34.2318 28 0.08035 1.85921 2014-04-01 16:00:31,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1158 16 9923 9907 34.216 16 17.6872 1.86235 2014-04-01 16:00:32,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1159 16 9927 9911 34.2003 16 4.46601 1.864 2014-04-01 16:00:33,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:00:33.473837min lat: 0.063861 max lat: 348.961 avg lat: 1.86371 2014-04-01 16:00:33,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:00:33,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1160 16 9929 9913 34.1777 8 0.529329 1.86371 2014-04-01 16:00:33,889.889 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:33,890.890 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:33,890.890 INFO:teuthology.task.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.009847 2014-04-01 16:00:33,890.890 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 1 0.00984697752247' 2014-04-01 16:00:34,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1161 16 9930 9914 34.1517 4 20.4687 1.86558 2014-04-01 16:00:34,842.842 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.1 primary-affinity to 0.00984698 (86452) 2014-04-01 16:00:35,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1162 16 9932 9916 34.1292 8 0.174246 1.86599 2014-04-01 16:00:36,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1163 16 9935 9919 34.1102 12 0.173797 1.86768 2014-04-01 16:00:37,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1164 16 9946 9930 34.1187 44 0.10608 1.86996 2014-04-01 16:00:38,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1165 16 9946 9930 34.0894 0 - 1.86996 2014-04-01 16:00:39,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1166 16 9946 9930 34.0602 0 - 1.86996 2014-04-01 16:00:39,856.856 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:39,857.857 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:39,857.857 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:00:39,858.858 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:00:40,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1167 16 9951 9935 34.0481 6.66667 7.25281 1.87066 2014-04-01 16:00:40,709.709 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:00:41,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1168 16 9956 9940 34.0361 20 0.109836 1.87299 2014-04-01 16:00:42,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1169 15 9958 9943 34.0172 12 9.8288 1.87455 2014-04-01 16:00:43,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1170 16 9986 9970 34.0805 108 0.541926 1.87731 2014-04-01 16:00:44,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1171 16 9996 9980 34.0855 40 0.360618 1.87628 2014-04-01 16:00:45,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1172 16 10001 9985 34.0735 20 0.187029 1.8755 2014-04-01 16:00:46,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1173 16 10001 9985 34.0444 0 - 1.8755 2014-04-01 16:00:46,633.633 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 5, 4, 3, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:46,634.634 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:46,634.634 INFO:teuthology.task.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 5, 4, 3, 2] 2014-04-01 16:00:46,634.634 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-04-01 16:00:47,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1174 16 10001 9985 34.0154 0 - 1.8755 2014-04-01 16:00:47,694.694 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.5. 2014-04-01 16:00:48,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1175 16 10007 9991 34.0069 8 0.231142 1.87587 2014-04-01 16:00:49,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1176 16 10019 10003 34.0188 48 0.776813 1.87753 2014-04-01 16:00:50,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1177 16 10023 10007 34.0035 16 0.261829 1.87696 2014-04-01 16:00:51,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1178 16 10023 10007 33.9746 0 - 1.87696 2014-04-01 16:00:52,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1179 16 10023 10007 33.9458 0 - 1.87696 2014-04-01 16:00:52,705.705 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:00:52,706.706 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:00:52,706.706 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:00:52,707.707 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:00:53,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:00:53.476320min lat: 0.063861 max lat: 348.961 avg lat: 1.87696 2014-04-01 16:00:53,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:00:53,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1180 16 10023 10007 33.917 0 - 1.87696 2014-04-01 16:00:53,889.889 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:00:54,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1181 16 10025 10009 33.8951 2 0.12026 1.87709 2014-04-01 16:00:54,850.850 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 16:00:54,851.851 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 46' 2014-04-01 16:00:55,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1182 16 10025 10009 33.8664 0 - 1.87709 2014-04-01 16:00:56,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1183 16 10033 10017 33.8648 16 8.08575 1.87921 2014-04-01 16:00:56,681.681 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 46 2014-04-01 16:00:57,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1184 16 10037 10021 33.8498 16 1.533 1.87957 2014-04-01 16:00:58,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1185 16 10044 10028 33.8448 28 0.437154 1.88062 2014-04-01 16:00:59,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1186 16 10045 10029 33.8197 4 1.0513 1.88054 2014-04-01 16:01:00,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1187 16 10046 10030 33.7945 4 2.18706 1.88057 2014-04-01 16:01:01,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1188 16 10046 10030 33.7661 0 - 1.88057 2014-04-01 16:01:01,695.695 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:01:01,696.696 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:01:01,697.697 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 16:01:01,697.697 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:01:02,497.497 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:01:02,850.850 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1189 16 10047 10031 33.741 2 3.03213 1.88068 2014-04-01 16:01:03,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1190 16 10047 10031 33.7127 0 - 1.88068 2014-04-01 16:01:04,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1191 16 10047 10031 33.6844 0 - 1.88068 2014-04-01 16:01:05,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1192 16 10047 10031 33.6561 0 - 1.88068 2014-04-01 16:01:06,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1193 16 10047 10031 33.6279 0 - 1.88068 2014-04-01 16:01:07,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1194 16 10047 10031 33.5998 0 - 1.88068 2014-04-01 16:01:08,473.473 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:01:08,474.474 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:01:08,474.474 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 16:01:08,475.475 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:01:08,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1195 16 10047 10031 33.5716 0 - 1.88068 2014-04-01 16:01:09,229.229 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:01:09,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1196 16 10047 10031 33.5436 0 - 1.88068 2014-04-01 16:01:10,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1197 16 10047 10031 33.5156 0 - 1.88068 2014-04-01 16:01:11,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1198 16 10047 10031 33.4876 0 - 1.88068 2014-04-01 16:01:12,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1199 16 10047 10031 33.4596 0 - 1.88068 2014-04-01 16:01:13,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:01:13.478782min lat: 0.063861 max lat: 348.961 avg lat: 1.88068 2014-04-01 16:01:13,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:01:13,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1200 16 10047 10031 33.4318 0 - 1.88068 2014-04-01 16:01:14,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1201 16 10049 10033 33.4106 0.666667 15.9391 1.88349 2014-04-01 16:01:15,001.001 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:01:15,002.002 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:01:15,002.002 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 16:01:15,003.003 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:01:15,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1202 16 10049 10033 33.3828 0 - 1.88349 2014-04-01 16:01:15,787.787 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:01:16,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1203 16 10049 10033 33.355 0 - 1.88349 2014-04-01 16:01:17,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1204 16 10049 10033 33.3273 0 - 1.88349 2014-04-01 16:01:18,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1205 16 10049 10033 33.2997 0 - 1.88349 2014-04-01 16:01:19,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1206 16 10049 10033 33.2721 0 - 1.88349 2014-04-01 16:01:20,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1207 16 10049 10033 33.2445 0 - 1.88349 2014-04-01 16:01:21,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1208 16 10049 10033 33.217 0 - 1.88349 2014-04-01 16:01:21,591.591 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 4, 3, 2] 2014-04-01 16:01:21,592.592 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:01:21,593.593 INFO:teuthology.task.thrashosds.thrasher:Killing osd 4, live_osds are [1, 0, 5, 4, 3, 2] 2014-04-01 16:01:21,594.594 DEBUG:teuthology.task.ceph.osd.4:waiting for process to exit 2014-04-01 16:01:21,855.855 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:01:21.770025 7f82a82f0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6805/4947 pipe(0x7f82a0019560 sd=12 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a00197c0).fault 2014-04-01 16:01:21,860.860 INFO:teuthology.task.ceph.osd.4:Stopped 2014-04-01 16:01:22,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1209 16 10050 10034 33.1928 0.5 21.5315 1.88544 2014-04-01 16:01:23,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1210 16 10050 10034 33.1654 0 - 1.88544 2014-04-01 16:01:24,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1211 16 10050 10034 33.138 0 - 1.88544 2014-04-01 16:01:25,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1212 16 10050 10034 33.1107 0 - 1.88544 2014-04-01 16:01:26,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1213 16 10050 10034 33.0834 0 - 1.88544 2014-04-01 16:01:26,858.858 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [4] live_osds: [1, 0, 5, 3, 2] 2014-04-01 16:01:26,859.859 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 4 2014-04-01 16:01:26,860.860 INFO:teuthology.task.ceph.osd.4:Restarting 2014-04-01 16:01:26,861.861 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4' 2014-04-01 16:01:27,183.183 INFO:teuthology.task.ceph.osd.4:Started 2014-04-01 16:01:27,184.184 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:27,369.369 INFO:teuthology.task.ceph.osd.4.out:[10.214.133.38]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal 2014-04-01 16:01:27,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1214 16 10050 10034 33.0561 0 - 1.88544 2014-04-01 16:01:27,611.611 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:27,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:27,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:27,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:27,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:27,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:27,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:27,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:27,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:27,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:27,617.617 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:27,617.617 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:27,617.617 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:27,848.848 INFO:teuthology.task.ceph.osd.4.err:[10.214.133.38]: 2014-04-01 07:01:27.761764 7fb87597c780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 16:01:28,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1215 16 10050 10034 33.0289 0 - 1.88544 2014-04-01 16:01:29,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1216 16 10050 10034 33.0017 0 - 1.88544 2014-04-01 16:01:30,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1217 16 10050 10034 32.9746 0 - 1.88544 2014-04-01 16:01:31,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1218 16 10050 10034 32.9476 0 - 1.88544 2014-04-01 16:01:32,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1219 16 10050 10034 32.9205 0 - 1.88544 2014-04-01 16:01:32,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:33,041.041 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:33,041.041 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:33,041.041 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:33,042.042 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:33,042.042 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:33,042.042 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:33,042.042 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:33,043.043 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:33,043.043 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:33,043.043 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:33,043.043 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:33,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:33,045.045 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:33,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:01:33.481469min lat: 0.063861 max lat: 348.961 avg lat: 1.88544 2014-04-01 16:01:33,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:01:33,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1220 16 10050 10034 32.8935 0 - 1.88544 2014-04-01 16:01:35,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1221 16 10050 10034 32.8666 0 - 1.88544 2014-04-01 16:01:35,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1222 16 10050 10034 32.8397 0 - 1.88544 2014-04-01 16:01:36,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1223 16 10050 10034 32.8129 0 - 1.88544 2014-04-01 16:01:37,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1224 16 10050 10034 32.7861 0 - 1.88544 2014-04-01 16:01:38,045.045 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:38,461.461 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:38,469.469 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:38,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:38,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:38,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:38,470.470 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:38,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1225 16 10050 10034 32.7593 0 - 1.88544 2014-04-01 16:01:39,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1226 16 10050 10034 32.7326 0 - 1.88544 2014-04-01 16:01:40,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1227 16 10050 10034 32.7059 0 - 1.88544 2014-04-01 16:01:41,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1228 16 10050 10034 32.6793 0 - 1.88544 2014-04-01 16:01:42,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1229 16 10050 10034 32.6527 0 - 1.88544 2014-04-01 16:01:43,468.468 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:43,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1230 16 10050 10034 32.6261 0 - 1.88544 2014-04-01 16:01:43,883.883 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:43,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:43,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:43,891.891 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:44,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1231 16 10050 10034 32.5996 0 - 1.88544 2014-04-01 16:01:45,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1232 16 10050 10034 32.5732 0 - 1.88544 2014-04-01 16:01:46,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1233 16 10053 10037 32.5565 0.5 0.422682 1.89301 2014-04-01 16:01:47,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1234 16 10053 10037 32.5301 0 - 1.89301 2014-04-01 16:01:48,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1235 16 10055 10039 32.5102 4 59.6016 1.90457 2014-04-01 16:01:48,891.891 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:49,315.315 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:49,324.324 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:49,324.324 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:49,325.325 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:49,325.325 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:49,325.325 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:49,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:49,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:49,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:49,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:49,327.327 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:49,327.327 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:49,327.327 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:49,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1236 16 10067 10051 32.5227 48 1.0364 1.93196 2014-04-01 16:01:50,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1237 16 10072 10056 32.5126 20 1.48405 1.93678 2014-04-01 16:01:51,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1238 16 10082 10066 32.5187 40 0.872153 1.9537 2014-04-01 16:01:52,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1239 16 10089 10073 32.515 28 1.25259 1.95995 2014-04-01 16:01:53,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:01:53.484148min lat: 0.063861 max lat: 348.961 avg lat: 1.96566 2014-04-01 16:01:53,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:01:53,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1240 16 10107 10091 32.5469 72 70.3535 1.96566 2014-04-01 16:01:54,322.322 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:01:54,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1241 16 10107 10091 32.5206 0 - 1.96566 2014-04-01 16:01:54,743.743 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:01:54,749.749 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:01:54,749.749 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:01:54,749.749 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:01:54,750.750 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:01:54,750.750 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:01:55,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1242 16 10109 10093 32.5009 4 2.01107 1.96573 2014-04-01 16:01:56,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1243 16 10116 10100 32.4973 28 0.972947 1.9663 2014-04-01 16:01:57,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1244 16 10120 10104 32.484 16 4.76445 1.9662 2014-04-01 16:01:58,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1245 16 10131 10115 32.4932 44 0.425406 1.96747 2014-04-01 16:01:59,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1246 16 10142 10126 32.5025 44 0.287109 1.96704 2014-04-01 16:01:59,749.749 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:02:00,191.191 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 16:02:00,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:00,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1247 16 10160 10144 32.5341 72 0.195478 1.96651 2014-04-01 16:02:00,680.680 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:02:00.593397 7ffc1c3fa700 0 -- :/1021113 >> 10.214.133.38:6789/0 pipe(0x7ffc180193f0 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7ffc18019650).fault 2014-04-01 16:02:01,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:01,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1248 16 10163 10147 32.5177 12 1.35316 1.96621 2014-04-01 16:02:02,223.223 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:02,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1249 16 10164 10148 32.4949 4 1.29232 1.96614 2014-04-01 16:02:03,028.028 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:03,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1250 16 10166 10150 32.4753 8 3.01433 1.96631 2014-04-01 16:02:03,832.832 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:04,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1251 16 10176 10160 32.4813 40 0.470047 1.96691 2014-04-01 16:02:05,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1252 16 10176 10160 32.4553 0 - 1.96691 2014-04-01 16:02:06,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1253 16 10181 10165 32.4454 10 6.35727 1.96756 2014-04-01 16:02:07,587.587 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:01:33.243652', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239716, u'health': u'HEALTH_OK', u'kb_used': 10089268, u'store_stats': {u'bytes_total': 6168711, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:01:33.960788', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437772436, u'health': u'HEALTH_OK', u'kb_used': 10556548, u'store_stats': {u'bytes_total': 7217009, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:01:33.492789', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239476, u'health': u'HEALTH_OK', u'kb_used': 10089508, u'store_stats': {u'bytes_total': 7217172, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs degraded'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12458/55807 objects degraded (22.323%); 285/18618 unfound (1.531%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 12458, u'num_pgs': 168, u'data_bytes': 77892423872, u'degraded_total': 55807, u'bytes_used': 54664687616, u'unfound_ratio': u'1.531', u'op_per_sec': 12, u'write_bytes_sec': 53519937, u'unfound_objects': 285, u'version': 683, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+degraded'}, {u'count': 21, u'state_name': u'active+recovery_wait'}, {u'count': 118, u'state_name': u'active+clean'}, {u'count': 21, u'state_name': u'active+recovering'}, {u'count': 2, u'state_name': u'active+recovering+remapped'}], u'degraded_ratio': u'22.323', u'bytes_avail': 2280848175104, u'unfound_total': 18618}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:07,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:07,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1254 16 10200 10184 32.4801 76 3.76418 1.96834 2014-04-01 16:02:08,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:08,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1255 16 10210 10194 32.4861 40 0.236137 1.96821 2014-04-01 16:02:09,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1256 16 10217 10201 32.4825 28 1.20604 1.96778 2014-04-01 16:02:10,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1257 16 10222 10206 32.4726 20 1.9767 1.96843 2014-04-01 16:02:11,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1258 16 10233 10217 32.4817 44 2.15883 1.9687 2014-04-01 16:02:12,180.180 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:12,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1259 16 10235 10219 32.4623 8 1.61445 1.9686 2014-04-01 16:02:12,935.935 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:13,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:02:13.487051min lat: 0.063861 max lat: 348.961 avg lat: 1.96848 2014-04-01 16:02:13,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:02:13,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1260 16 10237 10221 32.4429 8 2.1002 1.96848 2014-04-01 16:02:13,739.739 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:14,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1261 16 10253 10237 32.4679 64 3.13133 1.97042 2014-04-01 16:02:14,616.616 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:01:33.243652', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239716, u'health': u'HEALTH_OK', u'kb_used': 10089268, u'store_stats': {u'bytes_total': 6168711, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:01:33.960788', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437772436, u'health': u'HEALTH_OK', u'kb_used': 10556548, u'store_stats': {u'bytes_total': 7217009, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:01:33.492789', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239476, u'health': u'HEALTH_OK', u'kb_used': 10089508, u'store_stats': {u'bytes_total': 7217172, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9763/55939 objects degraded (17.453%); 285/18662 unfound (1.527%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 15498244, u'degraded_objects': 9763, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78076973248, u'degraded_total': 55939, u'bytes_used': 55048302592, u'unfound_ratio': u'1.527', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 47786252, u'op_per_sec': 11, u'unfound_objects': 285, u'version': 686, u'pgs_by_state': [{u'count': 24, u'state_name': u'active+recovery_wait'}, {u'count': 125, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.453', u'bytes_avail': 2280464560128, u'unfound_total': 18662}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:14,616.616 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:15,520.520 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:15,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1262 16 10266 10250 32.4834 52 0.917474 1.96893 2014-04-01 16:02:16,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1263 16 10266 10250 32.4576 0 - 1.96893 2014-04-01 16:02:17,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1264 16 10280 10264 32.4763 28 2.61762 1.96949 2014-04-01 16:02:18,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1265 16 10295 10279 32.498 60 1.61971 1.96872 2014-04-01 16:02:19,337.337 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:19,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1266 16 10299 10283 32.485 16 1.10606 1.96845 2014-04-01 16:02:20,087.087 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:20,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1267 16 10301 10285 32.4657 8 1.29053 1.96832 2014-04-01 16:02:20,894.894 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:21,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1268 16 10301 10285 32.4401 0 - 1.96832 2014-04-01 16:02:21,611.611 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:01:33.243652', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239716, u'health': u'HEALTH_OK', u'kb_used': 10089268, u'store_stats': {u'bytes_total': 6168711, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:01:33.960788', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437772436, u'health': u'HEALTH_OK', u'kb_used': 10556548, u'store_stats': {u'bytes_total': 7217009, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:01:33.492789', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239476, u'health': u'HEALTH_OK', u'kb_used': 10089508, u'store_stats': {u'bytes_total': 7217172, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9812/56083 objects degraded (17.495%); 285/18710 unfound (1.523%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6215675, u'degraded_objects': 9812, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78278299840, u'degraded_total': 56083, u'bytes_used': 55294930944, u'unfound_ratio': u'1.523', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 34807780, u'op_per_sec': 8, u'unfound_objects': 285, u'version': 690, u'pgs_by_state': [{u'count': 24, u'state_name': u'active+recovery_wait'}, {u'count': 126, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.495', u'bytes_avail': 2280217931776, u'unfound_total': 18710}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:21,613.613 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:22,308.308 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:22,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1269 16 10302 10286 32.4176 2 3.60284 1.96847 2014-04-01 16:02:23,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1270 16 10316 10300 32.4362 56 0.362715 1.97094 2014-04-01 16:02:24,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1271 16 10328 10312 32.4484 48 1.30478 1.97083 2014-04-01 16:02:25,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1272 16 10336 10320 32.4481 32 0.788403 1.97037 2014-04-01 16:02:26,087.087 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:26,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1273 16 10354 10338 32.4792 72 0.232645 1.96927 2014-04-01 16:02:26,831.831 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:27,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1274 16 10357 10341 32.4631 12 0.310532 1.96882 2014-04-01 16:02:27,624.624 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:28,350.350 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:01:33.243652', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239716, u'health': u'HEALTH_OK', u'kb_used': 10089268, u'store_stats': {u'bytes_total': 6168711, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:01:33.960788', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437772436, u'health': u'HEALTH_OK', u'kb_used': 10556548, u'store_stats': {u'bytes_total': 7217009, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:01:33.492789', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438239476, u'health': u'HEALTH_OK', u'kb_used': 10089508, u'store_stats': {u'bytes_total': 7217172, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9836/56308 objects degraded (17.468%); 285/18785 unfound (1.517%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16206814, u'degraded_objects': 9836, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78592872640, u'degraded_total': 56308, u'bytes_used': 55728824320, u'unfound_ratio': u'1.517', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 42649512, u'op_per_sec': 10, u'unfound_objects': 285, u'version': 693, u'pgs_by_state': [{u'count': 22, u'state_name': u'active+recovery_wait'}, {u'count': 129, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.468', u'bytes_avail': 2279784038400, u'unfound_total': 18785}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:28,351.351 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:28,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1275 16 10361 10345 32.4502 16 2.76446 1.96893 2014-04-01 16:02:29,062.062 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:29,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1276 16 10375 10359 32.4686 56 0.834781 1.96951 2014-04-01 16:02:30,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1277 16 10382 10366 32.4651 28 1.02513 1.96913 2014-04-01 16:02:31,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1278 16 10387 10371 32.4554 20 2.4269 1.96959 2014-04-01 16:02:32,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1279 16 10397 10381 32.4613 40 2.87417 1.97022 2014-04-01 16:02:32,994.994 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:33,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:02:33.489689min lat: 0.063861 max lat: 348.961 avg lat: 1.96974 2014-04-01 16:02:33,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:02:33,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1280 16 10407 10391 32.4671 40 0.260623 1.96974 2014-04-01 16:02:33,783.783 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:34,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1281 16 10411 10395 32.4543 16 2.8587 1.96972 2014-04-01 16:02:34,694.694 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:35,490.490 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9849/56440 objects degraded (17.450%); 285/18829 unfound (1.514%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10958965, u'degraded_objects': 9849, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78777422016, u'degraded_total': 56440, u'bytes_used': 56181149696, u'unfound_ratio': u'1.514', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 24353255, u'op_per_sec': 5, u'unfound_objects': 285, u'version': 697, u'pgs_by_state': [{u'count': 19, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.450', u'bytes_avail': 2279331713024, u'unfound_total': 18829}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:35,491.491 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:35,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1282 16 10413 10397 32.4352 8 1.65389 1.96975 2014-04-01 16:02:36,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:36,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1283 16 10417 10401 32.4224 16 0.269466 1.96949 2014-04-01 16:02:37,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1284 16 10417 10401 32.3971 0 - 1.96949 2014-04-01 16:02:38,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1285 16 10418 10402 32.375 2 6.10504 1.96988 2014-04-01 16:02:39,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1286 16 10422 10406 32.3623 16 2.72939 1.97053 2014-04-01 16:02:40,091.091 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:40,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1287 16 10440 10424 32.3931 72 8.13526 1.97388 2014-04-01 16:02:40,903.903 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:41,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1288 16 10449 10433 32.3959 36 1.53574 1.97413 2014-04-01 16:02:41,884.884 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:42,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1289 16 10459 10443 32.4018 40 1.65606 1.97359 2014-04-01 16:02:42,613.613 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9839/56575 objects degraded (17.391%); 285/18874 unfound (1.510%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6528362, u'degraded_objects': 9839, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78966165696, u'degraded_total': 56575, u'bytes_used': 56495828992, u'unfound_ratio': u'1.510', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 20517710, u'op_per_sec': 4, u'unfound_objects': 285, u'version': 701, u'pgs_by_state': [{u'count': 19, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.391', u'bytes_avail': 2279017033728, u'unfound_total': 18874}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:42,613.613 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:43,337.337 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:43,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1290 16 10466 10450 32.3984 28 1.73307 1.97328 2014-04-01 16:02:44,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1291 16 10467 10451 32.3764 4 1.91035 1.97328 2014-04-01 16:02:45,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1292 16 10477 10461 32.3823 40 4.64059 1.97373 2014-04-01 16:02:46,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1293 16 10479 10463 32.3634 8 4.41921 1.97382 2014-04-01 16:02:47,114.114 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:47,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1294 16 10482 10466 32.3477 12 5.06692 1.97432 2014-04-01 16:02:47,845.845 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:48,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1295 16 10489 10473 32.3443 28 3.76044 1.97593 2014-04-01 16:02:48,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:49,544.544 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9856/56710 objects degraded (17.380%); 285/18919 unfound (1.506%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9857938, u'degraded_objects': 9856, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79154909376, u'degraded_total': 56710, u'bytes_used': 56841166848, u'unfound_ratio': u'1.506', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 27109331, u'op_per_sec': 6, u'unfound_objects': 285, u'version': 706, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.380', u'bytes_avail': 2278671695872, u'unfound_total': 18919}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:49,545.545 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:49,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1296 16 10492 10476 32.3286 12 2.76956 1.97633 2014-04-01 16:02:50,252.252 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:50,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1297 16 10503 10487 32.3376 44 0.411356 1.97584 2014-04-01 16:02:51,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1298 16 10512 10496 32.3404 36 0.697775 1.97678 2014-04-01 16:02:52,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1299 16 10521 10505 32.3432 36 2.47911 1.97651 2014-04-01 16:02:53,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:02:53.492593min lat: 0.063861 max lat: 348.961 avg lat: 1.97667 2014-04-01 16:02:53,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:02:53,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1300 16 10529 10513 32.343 32 1.69023 1.97667 2014-04-01 16:02:54,028.028 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:54,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1301 16 10532 10516 32.3273 12 0.417835 1.9764 2014-04-01 16:02:54,697.697 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:55,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1302 16 10537 10521 32.3179 20 2.03918 1.97714 2014-04-01 16:02:55,634.634 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:02:56,424.424 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9879/56812 objects degraded (17.389%); 285/18953 unfound (1.504%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5712097, u'degraded_objects': 9879, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79297515712, u'degraded_total': 56812, u'bytes_used': 56983797760, u'unfound_ratio': u'1.504', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 38842262, u'op_per_sec': 9, u'unfound_objects': 285, u'version': 709, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.389', u'bytes_avail': 2278529064960, u'unfound_total': 18953}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:02:56,425.425 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:02:56,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1303 16 10540 10524 32.3023 12 0.357758 1.97716 2014-04-01 16:02:57,137.137 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:02:57,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1304 16 10543 10527 32.2867 12 5.05226 1.97781 2014-04-01 16:02:58,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1305 16 10547 10531 32.2742 16 4.89232 1.97859 2014-04-01 16:02:59,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1306 16 10549 10533 32.2556 8 3.43842 1.97859 2014-04-01 16:03:00,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1307 16 10549 10533 32.231 0 - 1.97859 2014-04-01 16:03:01,082.082 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:03:01,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1308 16 10551 10535 32.2124 4 6.25648 1.97957 2014-04-01 16:03:01,698.698 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:03:02,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1309 16 10553 10537 32.1939 8 4.8537 1.9805 2014-04-01 16:03:02,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:03:03,378.378 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9835/56905 objects degraded (17.283%); 285/18984 unfound (1.501%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 9835, u'num_pgs': 168, u'data_bytes': 79427539136, u'degraded_total': 56905, u'bytes_used': 57370173440, u'unfound_ratio': u'1.501', u'op_per_sec': 1, u'write_bytes_sec': 7741031, u'unfound_objects': 285, u'version': 713, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.283', u'bytes_avail': 2278142689280, u'unfound_total': 18984}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:03:03,379.379 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:03:03,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1310 16 10553 10537 32.1694 0 - 1.9805 2014-04-01 16:03:04,090.090 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:03:04,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1311 16 10558 10542 32.1601 10 10.7205 1.98272 2014-04-01 16:03:05,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1312 16 10559 10543 32.1386 4 0.298872 1.98256 2014-04-01 16:03:06,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1313 16 10563 10547 32.1263 16 0.47051 1.98291 2014-04-01 16:03:07,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1314 16 10567 10551 32.114 16 0.717839 1.98465 2014-04-01 16:03:08,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:03:08,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1315 16 10567 10551 32.0896 0 - 1.98465 2014-04-01 16:03:08,605.605 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:03:09,545.545 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:03:09,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1316 16 10571 10555 32.0774 8 2.43476 1.98596 2014-04-01 16:03:10,273.273 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:02:33.243860', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145796, u'health': u'HEALTH_OK', u'kb_used': 10183188, u'store_stats': {u'bytes_total': 8265863, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:02:33.961030', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684864, u'health': u'HEALTH_OK', u'kb_used': 10644120, u'store_stats': {u'bytes_total': 10362737, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:02:33.492992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438145468, u'health': u'HEALTH_OK', u'kb_used': 10183516, u'store_stats': {u'bytes_total': 10362900, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9799/56953 objects degraded (17.205%); 285/19000 unfound (1.500%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9278897, u'degraded_objects': 9799, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79494648000, u'degraded_total': 56953, u'bytes_used': 57556201472, u'unfound_ratio': u'1.500', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 14846236, u'op_per_sec': 3, u'unfound_objects': 285, u'version': 716, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 131, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.205', u'bytes_avail': 2277956661248, u'unfound_total': 19000}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:03:10,274.274 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:03:10,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1317 16 10576 10560 32.0682 20 0.543712 1.98771 2014-04-01 16:03:10,980.980 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:03:11,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1318 16 10582 10566 32.0621 24 14.7858 1.99027 2014-04-01 16:03:12,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1319 16 10593 10577 32.0711 44 0.640109 1.99415 2014-04-01 16:03:14,752.752 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:00,751.751 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:03:13.495274min lat: 0.063861 max lat: 348.961 avg lat: 1.99421 2014-04-01 16:04:00,751.751 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:00,751.751 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1320 16 10600 10584 32.0681 28 1.29555 1.99421 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1321 16 10607 10591 32.065 28 0.298936 1.99352 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1322 16 10614 10598 32.0619 28 0.209872 1.99388 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1323 16 10619 10603 32.0528 20 0.300415 1.9936 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1324 16 10622 10606 32.0376 12 0.296368 1.99325 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1325 16 10624 10608 32.0195 8 4.46865 1.99375 2014-04-01 16:04:00,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1326 16 10626 10610 32.0014 8 5.69054 1.99443 2014-04-01 16:04:00,910.910 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1327 16 10627 10611 31.9803 4 6.03915 1.99481 2014-04-01 16:04:00,910.910 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1328 16 10627 10611 31.9562 0 - 1.99481 2014-04-01 16:04:00,910.910 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1329 16 10631 10615 31.9442 8 8.50369 1.99636 2014-04-01 16:04:00,910.910 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1330 16 10631 10615 31.9202 0 - 1.99636 2014-04-01 16:04:00,910.910 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1331 16 10631 10615 31.8962 0 - 1.99636 2014-04-01 16:04:00,916.916 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1332 16 10632 10616 31.8752 1.33333 10.5821 1.99717 2014-04-01 16:04:00,917.917 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1333 16 10638 10622 31.8693 24 0.260673 1.99962 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1334 16 10639 10623 31.8484 4 11.479 2.00052 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1335 16 10641 10625 31.8306 8 1.82157 2.00043 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1336 16 10642 10626 31.8097 4 14.3729 2.00159 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1337 16 10644 10628 31.7919 8 11.756 2.00361 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1338 16 10644 10628 31.7682 0 - 2.00361 2014-04-01 16:04:00,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1339 16 10648 10632 31.7564 8 12.5943 2.0077 2014-04-01 16:04:00,919.919 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:03:33.497954min lat: 0.063861 max lat: 348.961 avg lat: 2.01019 2014-04-01 16:04:00,919.919 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:00,919.919 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1340 16 10651 10635 31.7416 12 7.80991 2.01019 2014-04-01 16:04:00,919.919 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1341 16 10659 10643 31.7418 32 0.433397 2.0131 2014-04-01 16:04:00,919.919 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1342 16 10664 10648 31.7331 20 3.48716 2.0135 2014-04-01 16:04:01,069.069 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1343 16 10675 10659 31.7422 44 0.251099 2.01311 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1344 16 10678 10662 31.7275 12 2.67207 2.0144 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1345 16 10678 10662 31.7039 0 - 2.0144 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1346 16 10678 10662 31.6804 0 - 2.0144 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1347 16 10678 10662 31.6569 0 - 2.0144 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1348 16 10697 10681 31.6897 19 4.89951 2.01908 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1349 16 10702 10686 31.6811 20 1.17994 2.01862 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1350 16 10708 10692 31.6754 24 1.57751 2.01832 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1351 16 10713 10697 31.6667 20 2.90136 2.01857 2014-04-01 16:04:01,070.070 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1352 16 10717 10701 31.6552 16 0.294046 2.01856 2014-04-01 16:04:01,071.071 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1353 16 10717 10701 31.6318 0 - 2.01856 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1354 16 10717 10701 31.6084 0 - 2.01856 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1355 16 10718 10702 31.588 1.33333 6.25168 2.01896 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1356 16 10723 10707 31.5795 20 6.1442 2.0212 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1357 16 10732 10716 31.5827 36 5.16203 2.02354 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1358 16 10736 10720 31.5713 16 3.26676 2.02447 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1359 16 10740 10724 31.5598 16 0.109931 2.02498 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:03:53.500727min lat: 0.063861 max lat: 348.961 avg lat: 2.02458 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:01,077.077 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1360 16 10751 10735 31.5689 44 0.625749 2.02458 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1361 16 10754 10738 31.5546 12 3.11157 2.02482 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1362 16 10763 10747 31.5578 36 3.67237 2.02548 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1363 16 10772 10756 31.5611 36 3.41721 2.02585 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1364 16 10784 10768 31.5731 48 1.47942 2.02578 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1365 16 10789 10773 31.5646 20 2.08629 2.02571 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1366 16 10801 10785 31.5767 48 2.08807 2.02567 2014-04-01 16:04:01,078.078 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1367 16 10809 10793 31.577 32 1.2513 2.02529 2014-04-01 16:04:01,732.732 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:02,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1368 16 10816 10800 31.5744 28 0.183183 2.02456 2014-04-01 16:04:03,046.046 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1369 16 10819 10803 31.5601 12 0.137174 2.02432 2014-04-01 16:04:03,886.886 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1370 16 10819 10803 31.537 0 - 2.02432 2014-04-01 16:04:04,024.024 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:04,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1371 16 10820 10804 31.5169 2 4.20759 2.02452 2014-04-01 16:04:05,051.051 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:03:33.244107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078904, u'health': u'HEALTH_OK', u'kb_used': 10250080, u'store_stats': {u'bytes_total': 10363015, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:03:33.961289', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437615348, u'health': u'HEALTH_OK', u'kb_used': 10713636, u'store_stats': {u'bytes_total': 13508465, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:03:33.493280', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078824, u'health': u'HEALTH_OK', u'kb_used': 10250160, u'store_stats': {u'bytes_total': 13508628, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9739/57706 objects degraded (16.877%); 285/19251 unfound (1.480%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5017759, u'degraded_objects': 9739, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80547418304, u'degraded_total': 57706, u'bytes_used': 59632353280, u'unfound_ratio': u'1.480', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 27597678, u'op_per_sec': 6, u'unfound_objects': 285, u'version': 745, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 133, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'16.877', u'bytes_avail': 2275880509440, u'unfound_total': 19251}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:05,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:05,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1372 16 10820 10804 31.494 0 - 2.02452 2014-04-01 16:04:06,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:06,707.707 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1373 16 10824 10808 31.4827 8 7.04406 2.02577 2014-04-01 16:04:07,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1374 16 10840 10824 31.5063 64 3.3777 2.03009 2014-04-01 16:04:08,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1375 16 10849 10833 31.5096 36 0.092441 2.0287 2014-04-01 16:04:09,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1376 16 10849 10833 31.4867 0 - 2.0287 2014-04-01 16:04:10,062.062 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:10,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1377 16 10857 10841 31.4871 16 0.262106 2.0297 2014-04-01 16:04:10,676.676 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:11,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1378 16 10861 10845 31.4758 16 3.85888 2.02951 2014-04-01 16:04:11,780.780 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:12,570.570 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:03:33.244107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078904, u'health': u'HEALTH_OK', u'kb_used': 10250080, u'store_stats': {u'bytes_total': 10363015, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:03:33.961289', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437615348, u'health': u'HEALTH_OK', u'kb_used': 10713636, u'store_stats': {u'bytes_total': 13508465, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:03:33.493280', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078824, u'health': u'HEALTH_OK', u'kb_used': 10250160, u'store_stats': {u'bytes_total': 13508628, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9278/57853 objects degraded (16.037%); 285/19300 unfound (1.477%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5555553, u'degraded_objects': 9278, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80752939200, u'degraded_total': 57853, u'bytes_used': 61071618048, u'unfound_ratio': u'1.477', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 34259246, u'op_per_sec': 8, u'unfound_objects': 285, u'version': 749, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'16.037', u'bytes_avail': 2274441244672, u'unfound_total': 19300}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:12,571.571 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:12,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1379 16 10862 10846 31.4559 4 0.297404 2.02935 2014-04-01 16:04:13,377.377 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:13,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:04:13.503648min lat: 0.063861 max lat: 348.961 avg lat: 2.02935 2014-04-01 16:04:13,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:13,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1380 16 10862 10846 31.4331 0 - 2.02935 2014-04-01 16:04:14,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1381 16 10868 10852 31.4277 12 4.26325 2.02989 2014-04-01 16:04:15,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1382 16 10869 10853 31.4079 4 0.226276 2.02972 2014-04-01 16:04:16,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1383 16 10869 10853 31.3852 0 - 2.02972 2014-04-01 16:04:17,220.220 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:17,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1384 16 10875 10859 31.3798 12 0.140506 2.03221 2014-04-01 16:04:17,976.976 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:18,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1385 16 10878 10862 31.3658 12 0.235273 2.03261 2014-04-01 16:04:18,806.806 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:19,510.510 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:03:33.244107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078904, u'health': u'HEALTH_OK', u'kb_used': 10250080, u'store_stats': {u'bytes_total': 10363015, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:03:33.961289', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437615348, u'health': u'HEALTH_OK', u'kb_used': 10713636, u'store_stats': {u'bytes_total': 13508465, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:03:33.493280', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078824, u'health': u'HEALTH_OK', u'kb_used': 10250160, u'store_stats': {u'bytes_total': 13508628, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9251/57889 objects degraded (15.981%); 285/19312 unfound (1.476%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 9251, u'num_pgs': 168, u'data_bytes': 80803270848, u'degraded_total': 57889, u'bytes_used': 61171159040, u'unfound_ratio': u'1.476', u'op_per_sec': 3, u'write_bytes_sec': 14244814, u'unfound_objects': 285, u'version': 752, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.981', u'bytes_avail': 2274341703680, u'unfound_total': 19312}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:19,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:19,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1386 16 10880 10864 31.349 8 10.53 2.03322 2014-04-01 16:04:20,210.210 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:20,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1387 16 10881 10865 31.3293 4 3.08815 2.03331 2014-04-01 16:04:21,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1388 16 10881 10865 31.3067 0 - 2.03331 2014-04-01 16:04:22,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1389 16 10881 10865 31.2841 0 - 2.03331 2014-04-01 16:04:23,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1390 16 10887 10871 31.2789 8 0.173806 2.03616 2014-04-01 16:04:24,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:24,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1391 16 10891 10875 31.2679 16 0.150166 2.03593 2014-04-01 16:04:24,692.692 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:25,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1392 16 10893 10877 31.2512 8 0.185301 2.0356 2014-04-01 16:04:25,660.660 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:26,438.438 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:03:33.244107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078904, u'health': u'HEALTH_OK', u'kb_used': 10250080, u'store_stats': {u'bytes_total': 10363015, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022967}}, {u'last_updated': u'2014-04-01 07:03:33.961289', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437615348, u'health': u'HEALTH_OK', u'kb_used': 10713636, u'store_stats': {u'bytes_total': 13508465, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:03:33.493280', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438078824, u'health': u'HEALTH_OK', u'kb_used': 10250160, u'store_stats': {u'bytes_total': 13508628, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022852}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9237/57931 objects degraded (15.945%); 285/19326 unfound (1.475%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4602339, u'degraded_objects': 9237, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80861991104, u'degraded_total': 57931, u'bytes_used': 61429829632, u'unfound_ratio': u'1.475', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 10125146, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 755, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.945', u'bytes_avail': 2274083033088, u'unfound_total': 19326}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:26,439.439 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:26,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1393 16 10896 10880 31.2374 12 6.03902 2.03706 2014-04-01 16:04:27,139.139 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:27,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1394 16 10907 10891 31.2465 44 9.64233 2.04373 2014-04-01 16:04:28,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1395 16 10908 10892 31.227 4 1.88281 2.04371 2014-04-01 16:04:29,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1396 16 10908 10892 31.2046 0 - 2.04371 2014-04-01 16:04:30,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1397 16 10908 10892 31.1823 0 - 2.04371 2014-04-01 16:04:31,074.074 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:31,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1398 16 10908 10892 31.16 0 - 2.04371 2014-04-01 16:04:31,841.841 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:32,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1399 16 10908 10892 31.1377 0 - 2.04371 2014-04-01 16:04:32,811.811 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:33,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:04:33.506590min lat: 0.063861 max lat: 348.961 avg lat: 2.04371 2014-04-01 16:04:33,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:33,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1400 16 10908 10892 31.1155 0 - 2.04371 2014-04-01 16:04:34,479.479 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:03:33.961289', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437615348, u'health': u'HEALTH_OK', u'kb_used': 10713636, u'store_stats': {u'bytes_total': 13508465, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3022689}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9210/58006 objects degraded (15.878%); 285/19351 unfound (1.473%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 9210, u'num_pgs': 168, u'data_bytes': 80966848704, u'degraded_total': 58006, u'bytes_used': 61556539392, u'unfound_ratio': u'1.473', u'op_per_sec': 5, u'write_bytes_sec': 23666260, u'unfound_objects': 285, u'version': 760, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.878', u'bytes_avail': 2273956323328, u'unfound_total': 19351}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:34,479.479 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:34,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1401 16 10908 10892 31.0933 0 - 2.04371 2014-04-01 16:04:35,204.204 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:35,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1402 16 10913 10897 31.0853 2.85714 0.312137 2.04699 2014-04-01 16:04:36,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1403 16 10923 10907 31.0917 40 9.24996 2.05138 2014-04-01 16:04:37,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1404 16 10932 10916 31.0952 36 0.073394 2.05106 2014-04-01 16:04:38,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1405 16 10935 10919 31.0816 12 0.267446 2.05249 2014-04-01 16:04:39,295.295 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:39,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1406 16 10939 10923 31.0709 16 0.173574 2.05183 2014-04-01 16:04:43,059.059 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1407 16 10939 10923 31.0488 0 - 2.05183 2014-04-01 16:04:43,059.059 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1408 16 10939 10923 31.0267 0 - 2.05183 2014-04-01 16:04:43,059.059 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1409 16 10939 10923 31.0047 0 - 2.05183 2014-04-01 16:04:43,592.592 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:44,215.215 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1410 16 10939 10923 30.9827 0 - 2.05183 2014-04-01 16:04:49,293.293 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1411 16 10944 10928 30.9749 4 9.48257 2.05671 2014-04-01 16:04:49,293.293 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1412 16 10946 10930 30.9587 8 0.469386 2.05641 2014-04-01 16:04:49,293.293 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1413 16 10950 10934 30.9481 16 0.211763 2.05985 2014-04-01 16:04:49,293.293 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1414 16 10950 10934 30.9262 0 - 2.05985 2014-04-01 16:04:49,293.293 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1415 16 10950 10934 30.9043 0 - 2.05985 2014-04-01 16:04:49,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1416 16 10950 10934 30.8825 0 - 2.05985 2014-04-01 16:04:49,643.643 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:50,353.353 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9144/58120 objects degraded (15.733%); 285/19389 unfound (1.470%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6158157, u'degraded_objects': 9144, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81126232256, u'degraded_total': 58120, u'bytes_used': 62075174912, u'unfound_ratio': u'1.470', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 11084683, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 770, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.733', u'bytes_avail': 2273437687808, u'unfound_total': 19389}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:50,354.354 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:50,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1417 16 10950 10934 30.8607 0 - 2.05985 2014-04-01 16:04:51,327.327 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:51,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1418 16 10950 10934 30.839 0 - 2.05985 2014-04-01 16:04:52,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1419 16 10950 10934 30.8172 0 - 2.05985 2014-04-01 16:04:53,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:04:53.509382min lat: 0.063861 max lat: 348.961 avg lat: 2.05985 2014-04-01 16:04:53,801.801 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:04:53,801.801 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1420 16 10950 10934 30.7955 0 - 2.05985 2014-04-01 16:04:54,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1421 16 10953 10937 30.7823 1.5 0.155912 2.06262 2014-04-01 16:04:55,246.246 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:55,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1422 16 10954 10938 30.7635 4 18.9538 2.06417 2014-04-01 16:04:55,804.804 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:56,596.596 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:04:56,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1423 16 10954 10938 30.7418 0 - 2.06417 2014-04-01 16:04:57,402.402 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9088/58132 objects degraded (15.633%); 285/19393 unfound (1.470%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7473691, u'degraded_objects': 9088, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81143009472, u'degraded_total': 58132, u'bytes_used': 62164652032, u'unfound_ratio': u'1.470', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 2802634, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 774, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.633', u'bytes_avail': 2273348210688, u'unfound_total': 19393}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:04:57,403.403 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:04:57,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1424 16 10954 10938 30.7203 0 - 2.06417 2014-04-01 16:04:58,097.097 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:04:58,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1425 16 10954 10938 30.6987 0 - 2.06417 2014-04-01 16:04:59,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1426 16 10954 10938 30.6772 0 - 2.06417 2014-04-01 16:05:00,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1427 16 10954 10938 30.6557 0 - 2.06417 2014-04-01 16:05:01,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1428 16 10954 10938 30.6342 0 - 2.06417 2014-04-01 16:05:01,875.875 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:02,528.528 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:02,842.842 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1429 16 10959 10943 30.6268 2.85714 0.092858 2.0716 2014-04-01 16:05:03,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:03,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1430 16 10964 10948 30.6193 20 26.1612 2.07311 2014-04-01 16:05:04,303.303 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9047/58135 objects degraded (15.562%); 285/19394 unfound (1.470%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 9047, u'num_pgs': 168, u'data_bytes': 81147203776, u'degraded_total': 58135, u'bytes_used': 62170963968, u'unfound_ratio': u'1.470', u'unfound_objects': 285, u'version': 777, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.562', u'bytes_avail': 2273341898752, u'unfound_total': 19394}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:04,303.303 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:04,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1431 16 10964 10948 30.5979 0 - 2.07311 2014-04-01 16:05:05,056.056 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:05,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1432 16 10965 10949 30.5794 2 28.0418 2.07548 2014-04-01 16:05:06,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1433 16 10965 10949 30.558 0 - 2.07548 2014-04-01 16:05:07,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1434 16 10965 10949 30.5367 0 - 2.07548 2014-04-01 16:05:08,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1435 16 10965 10949 30.5154 0 - 2.07548 2014-04-01 16:05:09,008.008 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:09,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1436 16 10965 10949 30.4942 0 - 2.07548 2014-04-01 16:05:09,790.790 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:10,568.568 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:10,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1437 16 10965 10949 30.473 0 - 2.07548 2014-04-01 16:05:11,310.310 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9030/58174 objects degraded (15.522%); 285/19407 unfound (1.469%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2788421, u'degraded_objects': 9030, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81201729728, u'degraded_total': 58174, u'bytes_used': 62390759424, u'unfound_ratio': u'1.469', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 11153686, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 780, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.522', u'bytes_avail': 2273122103296, u'unfound_total': 19407}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:11,311.311 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:11,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1438 16 10965 10949 30.4518 0 - 2.07548 2014-04-01 16:05:12,007.007 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:12,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1439 16 10965 10949 30.4306 0 - 2.07548 2014-04-01 16:05:13,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:05:13.512221min lat: 0.063861 max lat: 348.961 avg lat: 2.07548 2014-04-01 16:05:13,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:05:13,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1440 16 10965 10949 30.4095 0 - 2.07548 2014-04-01 16:05:14,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1441 16 10965 10949 30.3884 0 - 2.07548 2014-04-01 16:05:15,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1442 16 10965 10949 30.3673 0 - 2.07548 2014-04-01 16:05:15,784.784 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:16,489.489 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:16,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1443 16 10965 10949 30.3462 0 - 2.07548 2014-04-01 16:05:17,298.298 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:17,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1444 16 10965 10949 30.3252 0 - 2.07548 2014-04-01 16:05:18,055.055 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8956/58174 objects degraded (15.395%); 285/19407 unfound (1.469%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8956, u'num_pgs': 168, u'data_bytes': 81201729728, u'degraded_total': 58174, u'bytes_used': 62484508672, u'unfound_ratio': u'1.469', u'unfound_objects': 285, u'version': 785, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.395', u'bytes_avail': 2273028354048, u'unfound_total': 19407}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:18,055.055 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:18,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1445 16 10965 10949 30.3042 0 - 2.07548 2014-04-01 16:05:18,811.811 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:19,736.736 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1446 16 10970 10954 30.2971 1.42857 34.9999 2.08711 2014-04-01 16:05:20,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1447 16 10977 10961 30.2955 28 0.178731 2.09522 2014-04-01 16:05:21,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1448 16 10983 10967 30.2912 24 0.170301 2.10293 2014-04-01 16:05:22,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1449 16 10986 10970 30.2786 12 19.429 2.10589 2014-04-01 16:05:22,735.735 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:23,644.644 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:23,849.849 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1450 16 10988 10972 30.2632 8 19.8246 2.1092 2014-04-01 16:05:24,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1451 16 10988 10972 30.2423 0 - 2.1092 2014-04-01 16:05:24,832.832 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:25,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1452 16 10988 10972 30.2215 0 - 2.1092 2014-04-01 16:05:25,607.607 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8944/58222 objects degraded (15.362%); 285/19423 unfound (1.467%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4691203, u'degraded_objects': 8944, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81268838592, u'degraded_total': 58222, u'bytes_used': 62719795200, u'unfound_ratio': u'1.467', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 12509875, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 788, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 19, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.362', u'bytes_avail': 2272793067520, u'unfound_total': 19423}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:25,608.608 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:26,472.472 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:26,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1453 16 10988 10972 30.2007 0 - 2.1092 2014-04-01 16:05:27,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1454 16 10988 10972 30.1799 0 - 2.1092 2014-04-01 16:05:28,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1455 16 10988 10972 30.1592 0 - 2.1092 2014-04-01 16:05:29,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1456 16 10993 10977 30.1522 3.33333 0.266346 2.11092 2014-04-01 16:05:30,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:30,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1457 16 10993 10977 30.1315 0 - 2.11092 2014-04-01 16:05:31,180.180 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:31,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1458 16 10994 10978 30.1136 2 25.5109 2.11305 2014-04-01 16:05:31,979.979 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:32,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1459 16 11001 10985 30.1121 28 0.231172 2.11483 2014-04-01 16:05:32,791.791 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:04:33.244384', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438044256, u'health': u'HEALTH_OK', u'kb_used': 10284728, u'store_stats': {u'bytes_total': 4129660, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:04:33.961599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437552836, u'health': u'HEALTH_OK', u'kb_used': 10776148, u'store_stats': {u'bytes_total': 4654511, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:04:33.493572', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438043996, u'health': u'HEALTH_OK', u'kb_used': 10284988, u'store_stats': {u'bytes_total': 4653876, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8863/58246 objects degraded (15.216%); 285/19431 unfound (1.467%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8863, u'num_pgs': 168, u'data_bytes': 81302393024, u'degraded_total': 58246, u'bytes_used': 62873837568, u'unfound_ratio': u'1.467', u'op_per_sec': 0, u'write_bytes_sec': 3847773, u'unfound_objects': 285, u'version': 793, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.216', u'bytes_avail': 2272639025152, u'unfound_total': 19431}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:32,792.792 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:33,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:05:33.514769min lat: 0.063861 max lat: 348.961 avg lat: 2.11483 2014-04-01 16:05:33,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:05:33,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1460 16 11001 10985 30.0915 0 - 2.11483 2014-04-01 16:05:33,621.621 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:34,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1461 16 11005 10989 30.0819 8 0.115108 2.11673 2014-04-01 16:05:35,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1462 16 11005 10989 30.0613 0 - 2.11673 2014-04-01 16:05:36,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1463 16 11005 10989 30.0408 0 - 2.11673 2014-04-01 16:05:37,385.385 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:37,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1464 16 11005 10989 30.0202 0 - 2.11673 2014-04-01 16:05:38,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:38,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1465 16 11005 10989 29.9997 0 - 2.11673 2014-04-01 16:05:39,038.038 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:39,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1466 16 11005 10989 29.9793 0 - 2.11673 2014-04-01 16:05:39,824.824 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.016778', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.045958', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8856/58279 objects degraded (15.196%); 285/19442 unfound (1.466%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1285653, u'degraded_objects': 8856, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81348530368, u'degraded_total': 58279, u'bytes_used': 62958620672, u'unfound_ratio': u'1.466', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 11570880, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 796, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 18, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.196', u'bytes_avail': 2272554242048, u'unfound_total': 19442}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:39,825.825 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:40,548.548 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:40,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1467 16 11009 10993 29.9697 2.66667 20.5202 2.12123 2014-04-01 16:05:41,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1468 16 11009 10993 29.9493 0 - 2.12123 2014-04-01 16:05:42,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1469 16 11009 10993 29.9289 0 - 2.12123 2014-04-01 16:05:43,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1470 16 11009 10993 29.9086 0 - 2.12123 2014-04-01 16:05:44,320.320 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:44,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1471 16 11011 10995 29.8937 2 15.2828 2.12417 2014-04-01 16:05:44,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:45,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1472 16 11022 11006 29.9033 44 0.162199 2.12835 2014-04-01 16:05:46,019.019 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:46,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1473 16 11022 11006 29.883 0 - 2.12835 2014-04-01 16:05:46,735.735 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8808/58312 objects degraded (15.105%); 285/19453 unfound (1.465%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8457182, u'degraded_objects': 8808, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81394667712, u'degraded_total': 58312, u'bytes_used': 63082090496, u'unfound_ratio': u'1.465', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 8457182, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 800, u'pgs_by_state': [{u'count': 15, u'state_name': u'active+recovery_wait'}, {u'count': 136, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.105', u'bytes_avail': 2272430772224, u'unfound_total': 19453}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:46,736.736 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:47,446.446 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:47,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1474 16 11029 11013 29.8817 14 0.240277 2.13364 2014-04-01 16:05:48,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1475 16 11031 11015 29.8668 8 0.134024 2.13503 2014-04-01 16:05:49,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1476 16 11044 11028 29.8818 52 2.69996 2.13822 2014-04-01 16:05:50,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1477 16 11054 11038 29.8887 40 1.33264 2.13857 2014-04-01 16:05:51,257.257 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:51,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1478 16 11066 11050 29.9009 48 1.0001 2.13911 2014-04-01 16:05:51,914.914 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:52,846.846 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1479 16 11076 11060 29.9078 40 3.24034 2.13786 2014-04-01 16:05:53,017.017 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:05:54,141.141 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:05:53.517665min lat: 0.063861 max lat: 348.961 avg lat: 2.1375 2014-04-01 16:05:54,141.141 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:05:54,142.142 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1480 16 11078 11062 29.893 8 0.156096 2.1375 2014-04-01 16:05:54,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1481 16 11085 11069 29.8917 28 5.12123 2.13798 2014-04-01 16:05:54,739.739 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8812/58450 objects degraded (15.076%); 285/19499 unfound (1.462%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10087536, u'degraded_objects': 8812, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81587605696, u'degraded_total': 58450, u'bytes_used': 63413137408, u'unfound_ratio': u'1.462', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 33013754, u'op_per_sec': 7, u'unfound_objects': 285, u'version': 804, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.076', u'bytes_avail': 2272099725312, u'unfound_total': 19499}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:05:54,741.741 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:55,463.463 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:05:55,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1482 16 11087 11071 29.8769 8 0.422419 2.13804 2014-04-01 16:05:56,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1483 16 11087 11071 29.8568 0 - 2.13804 2014-04-01 16:05:57,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1484 16 11094 11078 29.8555 14 0.214601 2.13984 2014-04-01 16:05:58,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1485 16 11111 11095 29.8812 68 0.396212 2.14024 2014-04-01 16:05:59,398.398 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:05:59,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1486 16 11122 11106 29.8907 44 4.70055 2.1397 2014-04-01 16:06:00,196.196 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:00,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1487 16 11128 11112 29.8867 24 0.250953 2.13867 2014-04-01 16:06:01,131.131 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:01,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1488 16 11145 11129 29.9123 68 3.56838 2.1386 2014-04-01 16:06:01,881.881 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8852/58591 objects degraded (15.108%); 285/19546 unfound (1.458%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2158596, u'degraded_objects': 8852, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81784737984, u'degraded_total': 58591, u'bytes_used': 63681007616, u'unfound_ratio': u'1.458', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 29141052, u'op_per_sec': 6, u'unfound_objects': 285, u'version': 808, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.108', u'bytes_avail': 2271831855104, u'unfound_total': 19546}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:01,882.882 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:02,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1489 16 11155 11139 29.9191 40 0.15138 2.13718 2014-04-01 16:06:02,655.655 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:03,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1490 16 11169 11153 29.9366 56 1.52134 2.13696 2014-04-01 16:06:04,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1491 16 11185 11169 29.9594 64 0.298913 2.13519 2014-04-01 16:06:05,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1492 16 11193 11177 29.9608 32 0.63349 2.13385 2014-04-01 16:06:06,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:06,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1493 16 11200 11184 29.9595 28 0.126381 2.13283 2014-04-01 16:06:07,383.383 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:07,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1494 16 11200 11184 29.9394 0 - 2.13283 2014-04-01 16:06:08,185.185 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:08,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1495 16 11200 11184 29.9194 0 - 2.13283 2014-04-01 16:06:08,967.967 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8911/58861 objects degraded (15.139%); 285/19636 unfound (1.451%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8911, u'num_pgs': 168, u'data_bytes': 82162225344, u'degraded_total': 58861, u'bytes_used': 64740212736, u'unfound_ratio': u'1.451', u'op_per_sec': 23, u'write_bytes_sec': 97658074, u'unfound_objects': 285, u'version': 812, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.139', u'bytes_avail': 2270772649984, u'unfound_total': 19636}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:08,968.968 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:09,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1496 16 11206 11190 29.9154 8 3.91838 2.13471 2014-04-01 16:06:10,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:10,890.890 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1497 16 11213 11197 29.9142 28 0.065368 2.13408 2014-04-01 16:06:11,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1498 16 11218 11202 29.9075 20 6.9075 2.13642 2014-04-01 16:06:12,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1499 16 11220 11204 29.8929 8 0.283467 2.13609 2014-04-01 16:06:13,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:06:13.520725min lat: 0.063861 max lat: 348.961 avg lat: 2.13659 2014-04-01 16:06:13,776.776 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:06:13,776.776 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1500 16 11223 11207 29.881 12 0.199084 2.13659 2014-04-01 16:06:14,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1501 16 11238 11222 29.9011 60 0.213284 2.13717 2014-04-01 16:06:14,860.860 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:15,684.684 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:15,837.837 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1502 16 11244 11228 29.8971 24 0.105291 2.13734 2014-04-01 16:06:16,480.480 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:16,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1503 16 11245 11229 29.8799 4 0.664508 2.13721 2014-04-01 16:06:17,227.227 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8927/58987 objects degraded (15.134%); 285/19678 unfound (1.448%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1860434, u'degraded_objects': 8927, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82338386112, u'degraded_total': 58987, u'bytes_used': 64966082560, u'unfound_ratio': u'1.448', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 26976294, u'op_per_sec': 6, u'unfound_objects': 285, u'version': 817, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.134', u'bytes_avail': 2270546780160, u'unfound_total': 19678}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:17,228.228 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:17,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1504 16 11249 11233 29.8707 16 0.125859 2.13767 2014-04-01 16:06:18,046.046 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:18,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1505 16 11249 11233 29.8508 0 - 2.13767 2014-04-01 16:06:19,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1506 16 11249 11233 29.831 0 - 2.13767 2014-04-01 16:06:20,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1507 16 11255 11239 29.8271 8 5.9966 2.13862 2014-04-01 16:06:21,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1508 16 11260 11244 29.8206 20 10.257 2.13991 2014-04-01 16:06:21,968.968 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:22,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1509 16 11276 11260 29.8432 64 0.114852 2.13831 2014-04-01 16:06:23,009.009 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:23,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1510 16 11277 11261 29.8261 4 0.122518 2.13814 2014-04-01 16:06:24,278.278 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:24,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1511 16 11283 11267 29.8223 24 0.17273 2.13922 2014-04-01 16:06:25,288.288 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8921/59050 objects degraded (15.108%); 285/19699 unfound (1.447%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14821067, u'degraded_objects': 8921, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82426466496, u'degraded_total': 59050, u'bytes_used': 65220800512, u'unfound_ratio': u'1.447', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 13585978, u'op_per_sec': 3, u'unfound_objects': 285, u'version': 821, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.108', u'bytes_avail': 2270292062208, u'unfound_total': 19699}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:25,289.289 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:25,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1512 16 11285 11269 29.8078 8 5.08238 2.13971 2014-04-01 16:06:26,103.103 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:26,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1513 16 11289 11273 29.7987 16 12.0676 2.14006 2014-04-01 16:06:27,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1514 16 11289 11273 29.779 0 - 2.14006 2014-04-01 16:06:28,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1515 16 11289 11273 29.7594 0 - 2.14006 2014-04-01 16:06:29,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1516 16 11289 11273 29.7397 0 - 2.14006 2014-04-01 16:06:30,540.540 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:30,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1517 16 11289 11273 29.7201 0 - 2.14006 2014-04-01 16:06:31,322.322 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:31,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1518 16 11292 11276 29.7085 2.4 5.71217 2.14116 2014-04-01 16:06:32,366.366 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:32,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1519 16 11297 11281 29.7021 20 0.182903 2.14364 2014-04-01 16:06:33,143.143 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:05:33.244616', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004772, u'health': u'HEALTH_OK', u'kb_used': 10324212, u'store_stats': {u'bytes_total': 6751100, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605356}}, {u'last_updated': u'2014-04-01 07:05:33.961856', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437497080, u'health': u'HEALTH_OK', u'kb_used': 10831904, u'store_stats': {u'bytes_total': 7800239, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605919}}, {u'last_updated': u'2014-04-01 07:05:33.493888', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438004736, u'health': u'HEALTH_OK', u'kb_used': 10324248, u'store_stats': {u'bytes_total': 7799604, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3605284}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8909/59137 objects degraded (15.065%); 285/19728 unfound (1.445%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8909, u'num_pgs': 168, u'data_bytes': 82548101312, u'degraded_total': 59137, u'bytes_used': 65473593344, u'unfound_ratio': u'1.445', u'op_per_sec': 1, u'write_bytes_sec': 5114502, u'unfound_objects': 285, u'version': 825, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.065', u'bytes_avail': 2270039269376, u'unfound_total': 19728}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:33,144.144 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:33,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:06:33.523309min lat: 0.063783 max lat: 348.961 avg lat: 2.14855 2014-04-01 16:06:33,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:06:33,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1520 16 11305 11289 29.7036 32 0.127169 2.14855 2014-04-01 16:06:33,950.950 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:34,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1521 16 11310 11294 29.6972 20 1.35243 2.14845 2014-04-01 16:06:35,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1522 16 11315 11299 29.6908 20 0.168365 2.1501 2014-04-01 16:06:36,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1523 16 11322 11306 29.6897 28 0.125276 2.15258 2014-04-01 16:06:37,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1524 16 11324 11308 29.6755 8 0.176249 2.15223 2014-04-01 16:06:37,835.835 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:38,401.401 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:38,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1525 16 11324 11308 29.656 0 - 2.15223 2014-04-01 16:06:39,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:39,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1526 16 11325 11309 29.6392 2 7.85196 2.15273 2014-04-01 16:06:40,222.222 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8926/59227 objects degraded (15.071%); 285/19758 unfound (1.442%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9284225, u'degraded_objects': 8926, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82673930432, u'degraded_total': 59227, u'bytes_used': 65678647296, u'unfound_ratio': u'1.442', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 21353719, u'op_per_sec': 5, u'unfound_objects': 285, u'version': 828, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.071', u'bytes_avail': 2269834215424, u'unfound_total': 19758}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:40,223.223 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:40,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1527 16 11326 11310 29.6224 4 0.14265 2.15255 2014-04-01 16:06:40,981.981 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:41,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1528 16 11326 11310 29.603 0 - 2.15255 2014-04-01 16:06:42,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1529 16 11326 11310 29.5837 0 - 2.15255 2014-04-01 16:06:43,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1530 16 11330 11314 29.5748 5.33333 8.86764 2.15481 2014-04-01 16:06:44,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1531 16 11336 11320 29.5712 24 11.741 2.15573 2014-04-01 16:06:44,744.744 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:45,304.304 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:45,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1532 16 11338 11322 29.5571 8 10.4501 2.15712 2014-04-01 16:06:46,239.239 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:46,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1533 16 11344 11328 29.5534 24 13.4062 2.15863 2014-04-01 16:06:47,002.002 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8917/59248 objects degraded (15.050%); 285/19765 unfound (1.442%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8917, u'num_pgs': 168, u'data_bytes': 82703290560, u'degraded_total': 59248, u'bytes_used': 65789169664, u'unfound_ratio': u'1.442', u'op_per_sec': 0, u'write_bytes_sec': 3726330, u'unfound_objects': 285, u'version': 831, u'pgs_by_state': [{u'count': 14, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.050', u'bytes_avail': 2269723693056, u'unfound_total': 19765}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:47,003.003 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:47,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1534 16 11347 11331 29.542 12 0.103141 2.15925 2014-04-01 16:06:47,704.704 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:48,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1535 16 11349 11333 29.528 8 0.101587 2.15921 2014-04-01 16:06:49,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1536 16 11349 11333 29.5087 0 - 2.15921 2014-04-01 16:06:50,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1537 16 11349 11333 29.4895 0 - 2.15921 2014-04-01 16:06:51,513.513 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:51,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1538 16 11353 11337 29.4808 5.33333 0.10455 2.16134 2014-04-01 16:06:52,296.296 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:52,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1539 16 11354 11338 29.4642 4 6.66426 2.16174 2014-04-01 16:06:53,226.226 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:06:53,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:06:53.526108min lat: 0.063783 max lat: 348.961 avg lat: 2.16357 2014-04-01 16:06:53,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:06:53,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1540 16 11356 11340 29.4503 8 7.68025 2.16357 2014-04-01 16:06:54,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1541 16 11371 11355 29.4701 60 0.136672 2.16612 2014-04-01 16:06:55,707.707 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1542 16 11371 11355 29.451 0 - 2.16612 2014-04-01 16:06:56,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1543 16 11374 11358 29.4397 6 0.25415 2.16607 2014-04-01 16:06:56,954.954 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8848/59347 objects degraded (14.909%); 285/19798 unfound (1.440%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2415746, u'degraded_objects': 8848, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82841702592, u'degraded_total': 59347, u'bytes_used': 65726517248, u'unfound_ratio': u'1.440', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 12078733, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 838, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.909', u'bytes_avail': 2269786345472, u'unfound_total': 19798}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:06:56,955.955 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:06:57,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1544 16 11381 11365 29.4387 28 0.135177 2.16701 2014-04-01 16:06:57,663.663 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:06:58,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1545 16 11383 11367 29.4249 8 0.325611 2.16671 2014-04-01 16:06:59,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1546 16 11391 11375 29.4265 32 0.5519 2.16805 2014-04-01 16:07:00,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1547 16 11392 11376 29.4101 4 0.196859 2.16788 2014-04-01 16:07:01,607.607 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:01,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1548 16 11393 11377 29.3937 4 15.2227 2.16903 2014-04-01 16:07:02,380.380 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:02,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1549 16 11393 11377 29.3747 0 - 2.16903 2014-04-01 16:07:03,169.169 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:03,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1550 16 11403 11387 29.3815 20 0.306339 2.1718 2014-04-01 16:07:03,991.991 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8845/59443 objects degraded (14.880%); 285/19830 unfound (1.437%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8845, u'num_pgs': 168, u'data_bytes': 82975920320, u'degraded_total': 59443, u'bytes_used': 65971363840, u'unfound_ratio': u'1.437', u'op_per_sec': 5, u'write_bytes_sec': 22702961, u'unfound_objects': 285, u'version': 842, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.880', u'bytes_avail': 2269541498880, u'unfound_total': 19830}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:03,991.991 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:04,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1551 16 11407 11391 29.3729 16 1.10248 2.17375 2014-04-01 16:07:04,697.697 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:05,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1552 16 11407 11391 29.354 0 - 2.17375 2014-04-01 16:07:06,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1553 16 11407 11391 29.3351 0 - 2.17375 2014-04-01 16:07:07,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1554 16 11407 11391 29.3162 0 - 2.17375 2014-04-01 16:07:08,627.627 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:08,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1555 16 11407 11391 29.2974 0 - 2.17375 2014-04-01 16:07:09,345.345 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:09,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1556 16 11407 11391 29.2785 0 - 2.17375 2014-04-01 16:07:10,286.286 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:10,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1557 16 11407 11391 29.2597 0 - 2.17375 2014-04-01 16:07:11,051.051 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8841/59491 objects degraded (14.861%); 285/19846 unfound (1.436%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1878224, u'degraded_objects': 8841, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83043029184, u'degraded_total': 59491, u'bytes_used': 66097582080, u'unfound_ratio': u'1.436', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 12208460, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 845, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.861', u'bytes_avail': 2269415280640, u'unfound_total': 19846}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:11,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:11,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1558 16 11407 11391 29.2409 0 - 2.17375 2014-04-01 16:07:11,819.819 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:12,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1559 16 11415 11399 29.2427 4 0.154736 2.17477 2014-04-01 16:07:13,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:07:13.529023min lat: 0.063783 max lat: 348.961 avg lat: 2.17589 2014-04-01 16:07:13,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:07:13,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1560 16 11418 11402 29.2317 12 0.14371 2.17589 2014-04-01 16:07:14,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1561 16 11418 11402 29.2129 0 - 2.17589 2014-04-01 16:07:15,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1562 16 11418 11402 29.1942 0 - 2.17589 2014-04-01 16:07:15,765.765 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:16,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:16,764.764 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1563 16 11418 11402 29.1756 0 - 2.17589 2014-04-01 16:07:17,431.431 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:17,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1564 16 11418 11402 29.1569 0 - 2.17589 2014-04-01 16:07:18,216.216 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8792/59524 objects degraded (14.771%); 285/19857 unfound (1.435%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8792, u'num_pgs': 168, u'data_bytes': 83089166528, u'degraded_total': 59524, u'bytes_used': 66295848960, u'unfound_ratio': u'1.435', u'op_per_sec': 2, u'write_bytes_sec': 11738093, u'unfound_objects': 285, u'version': 850, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.771', u'bytes_avail': 2269217013760, u'unfound_total': 19857}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:18,216.216 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:18,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1565 16 11418 11402 29.1383 0 - 2.17589 2014-04-01 16:07:19,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:19,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1566 16 11418 11402 29.1197 0 - 2.17589 2014-04-01 16:07:20,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1567 16 11418 11402 29.1011 0 - 2.17589 2014-04-01 16:07:21,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1568 16 11418 11402 29.0825 0 - 2.17589 2014-04-01 16:07:22,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1569 16 11418 11402 29.064 0 - 2.17589 2014-04-01 16:07:23,138.138 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:23,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1570 16 11418 11402 29.0455 0 - 2.17589 2014-04-01 16:07:23,813.813 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:24,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1571 16 11419 11403 29.0295 0.363636 20.9598 2.17754 2014-04-01 16:07:24,794.794 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:25,527.527 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8753/59524 objects degraded (14.705%); 285/19857 unfound (1.435%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4743758, u'degraded_objects': 8753, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83089166528, u'degraded_total': 59524, u'bytes_used': 66411282432, u'unfound_ratio': u'1.435', u'recovering_objects_per_sec': 1, u'unfound_objects': 285, u'version': 854, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.705', u'bytes_avail': 2269101580288, u'unfound_total': 19857}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:25,528.528 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:25,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1572 16 11421 11405 29.0162 8 0.107674 2.1799 2014-04-01 16:07:26,398.398 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:26,705.705 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1573 16 11421 11405 28.9977 0 - 2.1799 2014-04-01 16:07:27,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1574 16 11421 11405 28.9793 0 - 2.1799 2014-04-01 16:07:28,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1575 16 11421 11405 28.9609 0 - 2.1799 2014-04-01 16:07:29,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1576 16 11431 11415 28.9679 10 0.112486 2.19035 2014-04-01 16:07:30,180.180 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:30,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1577 16 11435 11419 28.9597 16 0.067796 2.18965 2014-04-01 16:07:31,005.005 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:31,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1578 16 11435 11419 28.9413 0 - 2.18965 2014-04-01 16:07:31,837.837 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:32,529.529 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:06:33.244948', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437967292, u'health': u'HEALTH_OK', u'kb_used': 10361692, u'store_stats': {u'bytes_total': 5411901, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314733}}, {u'last_updated': u'2014-04-01 07:06:33.962099', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439304, u'health': u'HEALTH_OK', u'kb_used': 10889680, u'store_stats': {u'bytes_total': 5411986, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314818}}, {u'last_updated': u'2014-04-01 07:06:33.494148', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437966256, u'health': u'HEALTH_OK', u'kb_used': 10362728, u'store_stats': {u'bytes_total': 5411844, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3314676}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8711/59542 objects degraded (14.630%); 285/19863 unfound (1.435%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8744412, u'degraded_objects': 8711, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83114332352, u'degraded_total': 59542, u'bytes_used': 66475515904, u'unfound_ratio': u'1.435', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 3886405, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 858, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.630', u'bytes_avail': 2269037346816, u'unfound_total': 19863}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:32,530.530 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:32,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1579 16 11436 11420 28.9255 2 28.3628 2.19195 2014-04-01 16:07:33,231.231 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:33,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:07:33.531885min lat: 0.063783 max lat: 348.961 avg lat: 2.19179 2014-04-01 16:07:33,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:07:33,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1580 16 11437 11421 28.9097 4 0.426674 2.19179 2014-04-01 16:07:34,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1581 16 11438 11422 28.894 4 35.1727 2.19468 2014-04-01 16:07:35,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1582 16 11438 11422 28.8757 0 - 2.19468 2014-04-01 16:07:36,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1583 16 11438 11422 28.8575 0 - 2.19468 2014-04-01 16:07:37,006.006 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:37,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1584 16 11438 11422 28.8393 0 - 2.19468 2014-04-01 16:07:37,683.683 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:38,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:38,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1585 16 11438 11422 28.8211 0 - 2.19468 2014-04-01 16:07:39,404.404 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8676/59581 objects degraded (14.562%); 285/19876 unfound (1.434%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8676, u'num_pgs': 168, u'data_bytes': 83168858304, u'degraded_total': 59581, u'bytes_used': 66595479552, u'unfound_ratio': u'1.434', u'op_per_sec': 1, u'write_bytes_sec': 5483168, u'unfound_objects': 285, u'version': 862, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.562', u'bytes_avail': 2268917383168, u'unfound_total': 19876}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:39,405.405 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:39,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1586 16 11438 11422 28.8029 0 - 2.19468 2014-04-01 16:07:40,169.169 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:40,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1587 16 11438 11422 28.7847 0 - 2.19468 2014-04-01 16:07:41,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1588 16 11440 11424 28.7716 1.14286 28.5654 2.19785 2014-04-01 16:07:42,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1589 16 11441 11425 28.7561 4 43.1169 2.20144 2014-04-01 16:07:43,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1590 16 11441 11425 28.738 0 - 2.20144 2014-04-01 16:07:44,086.086 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:44,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1591 16 11441 11425 28.7199 0 - 2.20144 2014-04-01 16:07:44,719.719 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:45,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1592 16 11441 11425 28.7019 0 - 2.20144 2014-04-01 16:07:45,673.673 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:46,549.549 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8634/59590 objects degraded (14.489%); 285/19879 unfound (1.434%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2637223, u'degraded_objects': 8634, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83181441216, u'degraded_total': 59590, u'bytes_used': 66727473152, u'unfound_ratio': u'1.434', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 2637223, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 866, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.489', u'bytes_avail': 2268785389568, u'unfound_total': 19879}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:46,550.550 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:46,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1593 16 11441 11425 28.6838 0 - 2.20144 2014-04-01 16:07:48,128.128 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1594 16 11442 11426 28.6684 0.8 17.9652 2.20282 2014-04-01 16:07:48,394.394 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:48,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1595 16 11445 11429 28.6579 12 22.3365 2.20422 2014-04-01 16:07:49,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1596 16 11445 11429 28.64 0 - 2.20422 2014-04-01 16:07:50,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1597 16 11445 11429 28.622 0 - 2.20422 2014-04-01 16:07:51,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1598 16 11445 11429 28.6041 0 - 2.20422 2014-04-01 16:07:52,597.597 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:52,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1599 16 11445 11429 28.5862 0 - 2.20422 2014-04-01 16:07:53,413.413 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:53,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:07:53.534933min lat: 0.063783 max lat: 348.961 avg lat: 2.20422 2014-04-01 16:07:53,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:07:53,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1600 16 11445 11429 28.5684 0 - 2.20422 2014-04-01 16:07:54,371.371 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:07:54,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1601 16 11445 11429 28.5505 0 - 2.20422 2014-04-01 16:07:55,104.104 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8590/59599 objects degraded (14.413%); 285/19882 unfound (1.433%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5564018, u'degraded_objects': 8590, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83194024128, u'degraded_total': 59599, u'bytes_used': 66858430464, u'unfound_ratio': u'1.433', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 2782009, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 870, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.413', u'bytes_avail': 2268654432256, u'unfound_total': 19882}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:07:55,105.105 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:07:55,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1602 16 11446 11430 28.5352 0.571429 51.5756 2.20854 2014-04-01 16:07:55,854.854 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:07:56,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1603 16 11446 11430 28.5174 0 - 2.20854 2014-04-01 16:07:57,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1604 16 11450 11434 28.5096 8 15.7164 2.21224 2014-04-01 16:07:58,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1605 16 11452 11436 28.4968 8 0.154926 2.21189 2014-04-01 16:07:59,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1606 16 11458 11442 28.494 24 0.065662 2.21957 2014-04-01 16:07:59,730.730 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:00,288.288 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:00,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1607 16 11460 11444 28.4812 8 3.05861 2.21992 2014-04-01 16:08:01,394.394 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:01,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1608 16 11465 11449 28.476 20 0.259208 2.22356 2014-04-01 16:08:02,128.128 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8553/59620 objects degraded (14.346%); 285/19889 unfound (1.433%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4342062, u'degraded_objects': 8553, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83223384256, u'degraded_total': 59620, u'bytes_used': 66931728384, u'unfound_ratio': u'1.433', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 6513093, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 874, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.346', u'bytes_avail': 2268581134336, u'unfound_total': 19889}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:02,129.129 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:02,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1609 16 11465 11449 28.4583 0 - 2.22356 2014-04-01 16:08:02,893.893 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:03,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1610 16 11466 11450 28.4431 2 50.5213 2.22778 2014-04-01 16:08:04,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1611 16 11466 11450 28.4254 0 - 2.22778 2014-04-01 16:08:05,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1612 16 11466 11450 28.4078 0 - 2.22778 2014-04-01 16:08:06,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1613 16 11466 11450 28.3902 0 - 2.22778 2014-04-01 16:08:06,846.846 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:07,560.560 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:07,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1614 16 11468 11452 28.3775 2 0.127695 2.23116 2014-04-01 16:08:08,498.498 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:08,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1615 16 11473 11457 28.3724 20 0.204462 2.23161 2014-04-01 16:08:09,282.282 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8519/59653 objects degraded (14.281%); 285/19900 unfound (1.432%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8519, u'num_pgs': 168, u'data_bytes': 83269521600, u'degraded_total': 59653, u'bytes_used': 67727118336, u'unfound_ratio': u'1.432', u'op_per_sec': 0, u'write_bytes_sec': 1286952, u'unfound_objects': 285, u'version': 878, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.281', u'bytes_avail': 2267785744384, u'unfound_total': 19900}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:09,283.283 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:09,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1616 16 11473 11457 28.3548 0 - 2.23161 2014-04-01 16:08:10,099.099 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:10,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1617 16 11485 11469 28.3669 24 0.150791 2.24883 2014-04-01 16:08:11,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1618 16 11488 11472 28.3568 12 3.34551 2.24866 2014-04-01 16:08:12,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1619 16 11492 11476 28.3492 16 1.9355 2.24844 2014-04-01 16:08:13,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:08:13.538007min lat: 0.063783 max lat: 348.961 avg lat: 2.2515 2014-04-01 16:08:13,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:08:13,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1620 16 11497 11481 28.344 20 15.4342 2.2515 2014-04-01 16:08:13,886.886 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:14,675.675 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:14,837.837 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1621 16 11498 11482 28.329 4 0.609378 2.25136 2014-04-01 16:08:15,459.459 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:15,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1622 16 11498 11482 28.3116 0 - 2.25136 2014-04-01 16:08:16,220.220 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8524/59725 objects degraded (14.272%); 285/19924 unfound (1.430%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3585587, u'degraded_objects': 8524, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83370184896, u'degraded_total': 59725, u'bytes_used': 67892236288, u'unfound_ratio': u'1.430', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 21513523, u'op_per_sec': 5, u'unfound_objects': 285, u'version': 882, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.272', u'bytes_avail': 2267620626432, u'unfound_total': 19924}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:16,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:16,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1623 16 11498 11482 28.2941 0 - 2.25136 2014-04-01 16:08:16,923.923 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:17,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1624 16 11502 11486 28.2865 5.33333 0.069839 2.25156 2014-04-01 16:08:18,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1625 16 11506 11490 28.279 16 17.5491 2.25433 2014-04-01 16:08:19,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1626 16 11508 11492 28.2665 8 0.818958 2.25555 2014-04-01 16:08:20,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1627 16 11508 11492 28.2491 0 - 2.25555 2014-04-01 16:08:20,847.847 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:21,629.629 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:21,783.783 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1628 16 11508 11492 28.2318 0 - 2.25555 2014-04-01 16:08:22,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1629 16 11508 11492 28.2144 0 - 2.25555 2014-04-01 16:08:22,730.730 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:23,522.522 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8497/59788 objects degraded (14.212%); 285/19945 unfound (1.429%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8497, u'num_pgs': 168, u'data_bytes': 83458265280, u'degraded_total': 59788, u'bytes_used': 68085739520, u'unfound_ratio': u'1.429', u'op_per_sec': 3, u'write_bytes_sec': 12926933, u'unfound_objects': 285, u'version': 887, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.212', u'bytes_avail': 2267427123200, u'unfound_total': 19945}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:23,523.523 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:23,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1630 16 11516 11500 28.2168 8 0.086413 2.25651 2014-04-01 16:08:24,236.236 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:24,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1631 16 11523 11507 28.2166 28 10.6358 2.26417 2014-04-01 16:08:25,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1632 16 11528 11512 28.2116 20 0.163644 2.265 2014-04-01 16:08:26,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1633 16 11528 11512 28.1943 0 - 2.265 2014-04-01 16:08:27,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1634 16 11528 11512 28.1771 0 - 2.265 2014-04-01 16:08:28,314.314 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:28,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1635 16 11528 11512 28.1598 0 - 2.265 2014-04-01 16:08:29,028.028 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:29,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1636 16 11528 11512 28.1426 0 - 2.265 2014-04-01 16:08:29,965.965 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:30,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1637 16 11528 11512 28.1254 0 - 2.265 2014-04-01 16:08:30,736.736 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:07:33.245203', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937560, u'health': u'HEALTH_OK', u'kb_used': 10391424, u'store_stats': {u'bytes_total': 3419948, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:07:33.962331', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396468, u'health': u'HEALTH_OK', u'kb_used': 10932516, u'store_stats': {u'bytes_total': 3419863, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:07:33.494404', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437937436, u'health': u'HEALTH_OK', u'kb_used': 10391548, u'store_stats': {u'bytes_total': 3419756, u'bytes_log': 196608, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8500/59854 objects degraded (14.201%); 285/19967 unfound (1.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8779346, u'degraded_objects': 8500, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83550539968, u'degraded_total': 59854, u'bytes_used': 68258029568, u'unfound_ratio': u'1.427', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 18812885, u'op_per_sec': 4, u'unfound_objects': 285, u'version': 891, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.201', u'bytes_avail': 2267254833152, u'unfound_total': 19967}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:30,737.737 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:31,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:31,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1638 16 11528 11512 28.1083 0 - 2.265 2014-04-01 16:08:32,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1639 16 11528 11512 28.0911 0 - 2.265 2014-04-01 16:08:33,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:08:33.540841min lat: 0.063783 max lat: 348.961 avg lat: 2.26619 2014-04-01 16:08:33,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:08:33,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1640 16 11529 11513 28.0764 0.5 15.9637 2.26619 2014-04-01 16:08:34,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1641 16 11533 11517 28.0691 16 0.104441 2.26544 2014-04-01 16:08:35,237.237 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:35,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1642 16 11533 11517 28.052 0 - 2.26544 2014-04-01 16:08:36,013.013 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:36,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1643 16 11533 11517 28.0349 0 - 2.26544 2014-04-01 16:08:36,973.973 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:37,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1644 16 11533 11517 28.0178 0 - 2.26544 2014-04-01 16:08:37,700.700 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8461/59878 objects degraded (14.130%); 285/19975 unfound (1.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 8461, u'num_pgs': 168, u'data_bytes': 83584094400, u'degraded_total': 59878, u'bytes_used': 68352282624, u'unfound_ratio': u'1.427', u'op_per_sec': 2, u'write_bytes_sec': 9112988, u'unfound_objects': 285, u'version': 896, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.130', u'bytes_avail': 2267160580096, u'unfound_total': 19975}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:37,701.701 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:38,427.427 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:38,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1645 16 11533 11517 28.0008 0 - 2.26544 2014-04-01 16:08:39,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1646 16 11533 11517 27.9838 0 - 2.26544 2014-04-01 16:08:40,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1647 16 11533 11517 27.9668 0 - 2.26544 2014-04-01 16:08:41,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1648 16 11533 11517 27.9498 0 - 2.26544 2014-04-01 16:08:42,354.354 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:42,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1649 16 11533 11517 27.9329 0 - 2.26544 2014-04-01 16:08:43,066.066 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:43,776.776 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1650 16 11533 11517 27.916 0 - 2.26544 2014-04-01 16:08:43,996.996 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:44,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1651 16 11533 11517 27.899 0 - 2.26544 2014-04-01 16:08:44,721.721 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8416/59878 objects degraded (14.055%); 285/19975 unfound (1.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16698901, u'degraded_objects': 8416, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83584094400, u'degraded_total': 59878, u'bytes_used': 68494249984, u'unfound_ratio': u'1.427', u'recovering_objects_per_sec': 3, u'unfound_objects': 285, u'version': 900, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.055', u'bytes_avail': 2267018612736, u'unfound_total': 19975}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:44,722.722 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:45,502.502 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:45,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1652 16 11533 11517 27.8822 0 - 2.26544 2014-04-01 16:08:46,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1653 16 11533 11517 27.8653 0 - 2.26544 2014-04-01 16:08:47,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1654 16 11533 11517 27.8484 0 - 2.26544 2014-04-01 16:08:48,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1655 16 11534 11518 27.834 0.285714 25.6465 2.26747 2014-04-01 16:08:49,421.421 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:49,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1656 16 11534 11518 27.8172 0 - 2.26747 2014-04-01 16:08:50,203.203 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:50,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1657 16 11534 11518 27.8004 0 - 2.26747 2014-04-01 16:08:51,182.182 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:51,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1658 16 11534 11518 27.7837 0 - 2.26747 2014-04-01 16:08:52,022.022 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8390/59878 objects degraded (14.012%); 285/19975 unfound (1.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3760228, u'degraded_objects': 8390, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83584094400, u'degraded_total': 59878, u'bytes_used': 68534820864, u'unfound_ratio': u'1.427', u'recovering_objects_per_sec': 0, u'unfound_objects': 285, u'version': 904, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.012', u'bytes_avail': 2266978041856, u'unfound_total': 19975}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:08:52,023.023 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:52,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1659 16 11534 11518 27.7669 0 - 2.26747 2014-04-01 16:08:52,827.827 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:53,887.887 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:08:53.543517min lat: 0.063783 max lat: 348.961 avg lat: 2.26747 2014-04-01 16:08:53,888.888 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:08:53,888.888 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1660 16 11534 11518 27.7502 0 - 2.26747 2014-04-01 16:08:54,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1661 16 11534 11518 27.7335 0 - 2.26747 2014-04-01 16:08:55,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1662 16 11534 11518 27.7168 0 - 2.26747 2014-04-01 16:08:56,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1663 16 11534 11518 27.7001 0 - 2.26747 2014-04-01 16:08:56,884.884 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:08:57,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:08:57,751.751 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1664 16 11534 11518 27.6835 0 - 2.26747 2014-04-01 16:08:59,247.247 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1665 16 11534 11518 27.6669 0 - 2.26747 2014-04-01 16:08:59,425.425 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:08:59,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1666 16 11536 11520 27.6551 0.727273 36.3827 2.2734 2014-04-01 16:09:00,109.109 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8326/59881 objects degraded (13.904%); 285/19976 unfound (1.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8979613, u'degraded_objects': 8326, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83588288704, u'degraded_total': 59881, u'bytes_used': 68636475392, u'unfound_ratio': u'1.427', u'recovering_objects_per_sec': 2, u'unfound_objects': 285, u'version': 909, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.904', u'bytes_avail': 2266876387328, u'unfound_total': 19976}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:00,110.110 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:00,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1667 16 11539 11523 27.6457 12 0.163961 2.2759 2014-04-01 16:09:00,919.919 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:01,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1668 16 11542 11526 27.6363 12 0.110215 2.27907 2014-04-01 16:09:02,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1669 16 11542 11526 27.6197 0 - 2.27907 2014-04-01 16:09:03,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1670 16 11542 11526 27.6032 0 - 2.27907 2014-04-01 16:09:04,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1671 16 11542 11526 27.5867 0 - 2.27907 2014-04-01 16:09:04,820.820 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:05,522.522 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:05,923.923 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1672 16 11542 11526 27.5702 0 - 2.27907 2014-04-01 16:09:06,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1673 16 11542 11526 27.5537 0 - 2.27907 2014-04-01 16:09:06,705.705 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:07,602.602 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'29 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8164/59911 objects degraded (13.627%); 285/19986 unfound (1.426%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2886825, u'degraded_objects': 8164, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83630231744, u'degraded_total': 59911, u'bytes_used': 68737957888, u'unfound_ratio': u'1.426', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 4811375, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 913, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 139, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.627', u'bytes_avail': 2266774904832, u'unfound_total': 19986}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:07,603.603 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:07,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1674 16 11542 11526 27.5372 0 - 2.27907 2014-04-01 16:09:08,454.454 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:08,794.794 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1675 16 11542 11526 27.5208 0 - 2.27907 2014-04-01 16:09:09,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1676 16 11542 11526 27.5044 0 - 2.27907 2014-04-01 16:09:10,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1677 16 11542 11526 27.488 0 - 2.27907 2014-04-01 16:09:11,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1678 16 11542 11526 27.4716 0 - 2.27907 2014-04-01 16:09:12,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1679 16 11542 11526 27.4552 0 - 2.27907 2014-04-01 16:09:12,647.647 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:13,446.446 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:13,780.780 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:09:13.546273min lat: 0.063783 max lat: 348.961 avg lat: 2.27907 2014-04-01 16:09:13,781.781 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:09:13,781.781 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1680 16 11542 11526 27.4389 0 - 2.27907 2014-04-01 16:09:14,463.463 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:14,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1681 16 11543 11527 27.4249 0.307692 50.0627 2.28322 2014-04-01 16:09:15,217.217 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8070/59911 objects degraded (13.470%); 285/19986 unfound (1.426%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5881913, u'degraded_objects': 8070, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83630231744, u'degraded_total': 59911, u'bytes_used': 68841517056, u'unfound_ratio': u'1.426', u'recovering_objects_per_sec': 1, u'unfound_objects': 285, u'version': 917, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 140, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.470', u'bytes_avail': 2266671345664, u'unfound_total': 19986}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:15,218.218 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:15,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1682 16 11549 11533 27.4229 24 52.5358 2.30019 2014-04-01 16:09:16,010.010 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:16,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1683 16 11550 11534 27.409 4 16.6335 2.30143 2014-04-01 16:09:17,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1684 16 11550 11534 27.3927 0 - 2.30143 2014-04-01 16:09:18,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1685 16 11554 11538 27.3859 8 53.6649 2.31046 2014-04-01 16:09:19,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1686 16 11557 11541 27.3768 12 0.230014 2.30993 2014-04-01 16:09:19,926.926 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:20,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1687 16 11562 11546 27.3724 20 0.071789 2.31853 2014-04-01 16:09:20,850.850 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:21,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1688 16 11563 11547 27.3586 4 6.2349 2.31887 2014-04-01 16:09:21,794.794 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:22,583.583 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7202/59947 objects degraded (12.014%); 285/19998 unfound (1.425%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7473868, u'degraded_objects': 7202, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83680563392, u'degraded_total': 59947, u'bytes_used': 68932005888, u'unfound_ratio': u'1.425', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 8541563, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 921, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.014', u'bytes_avail': 2266580856832, u'unfound_total': 19998}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:22,584.584 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:22,627.627 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1689 16 11565 11549 27.3471 8 0.065212 2.32343 2014-04-01 16:09:23,350.350 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:23,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1690 16 11565 11549 27.331 0 - 2.32343 2014-04-01 16:09:24,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1691 16 11565 11549 27.3148 0 - 2.32343 2014-04-01 16:09:25,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1692 16 11569 11553 27.3081 5.33333 0.159792 2.32411 2014-04-01 16:09:26,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1693 16 11575 11559 27.3061 24 0.220066 2.32808 2014-04-01 16:09:27,279.279 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:27,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1694 16 11575 11559 27.29 0 - 2.32808 2014-04-01 16:09:27,998.998 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:28,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1695 16 11575 11559 27.2739 0 - 2.32808 2014-04-01 16:09:28,923.923 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:29,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1696 16 11575 11559 27.2578 0 - 2.32808 2014-04-01 16:09:29,709.709 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:08:33.245525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902312, u'health': u'HEALTH_OK', u'kb_used': 10426672, u'store_stats': {u'bytes_total': 5254956, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:08:33.962598', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437352348, u'health': u'HEALTH_OK', u'kb_used': 10976636, u'store_stats': {u'bytes_total': 6303447, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:08:33.494690', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437902196, u'health': u'HEALTH_OK', u'kb_used': 10426788, u'store_stats': {u'bytes_total': 6303340, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7176/60004 objects degraded (11.959%); 285/20017 unfound (1.424%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3649469, u'degraded_objects': 7176, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83760255168, u'degraded_total': 60004, u'bytes_used': 68629450752, u'unfound_ratio': u'1.424', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 8515429, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 926, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.959', u'bytes_avail': 2266883411968, u'unfound_total': 20017}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:29,710.710 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:30,474.474 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:30,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1697 16 11575 11559 27.2418 0 - 2.32808 2014-04-01 16:09:31,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1698 16 11575 11559 27.2257 0 - 2.32808 2014-04-01 16:09:32,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1699 16 11575 11559 27.2097 0 - 2.32808 2014-04-01 16:09:33,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:09:33.548907min lat: 0.063783 max lat: 348.961 avg lat: 2.32808 2014-04-01 16:09:33,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:09:33,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1700 16 11575 11559 27.1937 0 - 2.32808 2014-04-01 16:09:34,452.452 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:34,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1701 16 11575 11559 27.1777 0 - 2.32808 2014-04-01 16:09:35,421.421 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:35,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1702 16 11576 11560 27.1641 0.444444 20.2313 2.32963 2014-04-01 16:09:36,196.196 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:36,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1703 16 11577 11561 27.1505 4 0.223638 2.32945 2014-04-01 16:09:36,984.984 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7132/60016 objects degraded (11.883%); 285/20021 unfound (1.424%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10028878, u'degraded_objects': 7132, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83777032384, u'degraded_total': 60016, u'bytes_used': 68697137152, u'unfound_ratio': u'1.424', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 1253609, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 930, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.883', u'bytes_avail': 2266815725568, u'unfound_total': 20021}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:36,985.985 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:37,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1704 16 11577 11561 27.1346 0 - 2.32945 2014-04-01 16:09:37,696.696 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:38,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1705 16 11577 11561 27.1187 0 - 2.32945 2014-04-01 16:09:39,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1706 16 11577 11561 27.1028 0 - 2.32945 2014-04-01 16:09:40,627.627 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1707 16 11577 11561 27.0869 0 - 2.32945 2014-04-01 16:09:41,621.621 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:41,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1708 16 11577 11561 27.071 0 - 2.32945 2014-04-01 16:09:42,248.248 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:42,870.870 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1709 16 11580 11564 27.0622 2 0.123295 2.33538 2014-04-01 16:09:43,359.359 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:43,627.627 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1710 16 11582 11566 27.0511 8 0.108607 2.3365 2014-04-01 16:09:44,082.082 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7090/60019 objects degraded (11.813%); 285/20022 unfound (1.423%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 7090, u'num_pgs': 168, u'data_bytes': 83781226688, u'degraded_total': 60019, u'bytes_used': 68776636416, u'unfound_ratio': u'1.423', u'unfound_objects': 285, u'version': 934, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.813', u'bytes_avail': 2266736226304, u'unfound_total': 20022}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:44,083.083 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:44,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1711 16 11582 11566 27.0352 0 - 2.3365 2014-04-01 16:09:44,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:45,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1712 16 11582 11566 27.0195 0 - 2.3365 2014-04-01 16:09:46,645.645 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1713 16 11582 11566 27.0037 0 - 2.3365 2014-04-01 16:09:47,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1714 16 11582 11566 26.9879 0 - 2.3365 2014-04-01 16:09:48,576.576 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:48,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1715 16 11582 11566 26.9722 0 - 2.3365 2014-04-01 16:09:49,312.312 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:49,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1716 16 11582 11566 26.9565 0 - 2.3365 2014-04-01 16:09:50,268.268 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:50,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1717 16 11582 11566 26.9408 0 - 2.3365 2014-04-01 16:09:50,969.969 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7066/60034 objects degraded (11.770%); 285/20027 unfound (1.423%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9277181, u'degraded_objects': 7066, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83802198208, u'degraded_total': 60034, u'bytes_used': 68911755264, u'unfound_ratio': u'1.423', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 2319295, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 938, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.770', u'bytes_avail': 2266601107456, u'unfound_total': 20027}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:50,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:51,544.544 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:09:51.467431 7f835236d700 0 monclient: hunting for new mon 2014-04-01 16:09:51,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1718 16 11582 11566 26.9251 0 - 2.3365 2014-04-01 16:09:51,706.706 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:52,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1719 16 11582 11566 26.9094 0 - 2.3365 2014-04-01 16:09:53,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:09:53.551689min lat: 0.063783 max lat: 348.961 avg lat: 2.3365 2014-04-01 16:09:53,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:09:53,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1720 16 11582 11566 26.8938 0 - 2.3365 2014-04-01 16:09:54,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1721 16 11589 11573 26.8944 2.54545 36.0611 2.35961 2014-04-01 16:09:55,633.633 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:55,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1722 16 11600 11584 26.9044 44 0.098594 2.36068 2014-04-01 16:09:56,345.345 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:56,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1723 16 11601 11585 26.8911 4 30.7799 2.36313 2014-04-01 16:09:57,299.299 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:09:57,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1724 16 11601 11585 26.8755 0 - 2.36313 2014-04-01 16:09:58,068.068 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7018/60073 objects degraded (11.682%); 285/20040 unfound (1.422%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 7018, u'num_pgs': 168, u'data_bytes': 83856724160, u'degraded_total': 60073, u'bytes_used': 68971786240, u'unfound_ratio': u'1.422', u'op_per_sec': 3, u'write_bytes_sec': 15527864, u'unfound_objects': 285, u'version': 942, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.682', u'bytes_avail': 2266541076480, u'unfound_total': 20040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:09:58,069.069 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:09:58,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1725 16 11601 11585 26.8599 0 - 2.36313 2014-04-01 16:09:58,775.775 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:09:59,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1726 16 11601 11585 26.8443 0 - 2.36313 2014-04-01 16:10:00,737.737 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1727 16 11601 11585 26.8288 0 - 2.36313 2014-04-01 16:10:01,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1728 16 11601 11585 26.8133 0 - 2.36313 2014-04-01 16:10:02,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1729 16 11601 11585 26.7977 0 - 2.36313 2014-04-01 16:10:02,724.724 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:03,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1730 16 11601 11585 26.7823 0 - 2.36313 2014-04-01 16:10:04,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1731 16 11601 11585 26.7668 0 - 2.36313 2014-04-01 16:10:05,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1732 16 11607 11591 26.7652 2.66667 0.294689 2.36765 2014-04-01 16:10:06,291.291 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:06,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1733 16 11612 11596 26.7613 20 0.124716 2.36675 2014-04-01 16:10:07,263.263 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:07,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1734 16 11612 11596 26.7458 0 - 2.36675 2014-04-01 16:10:08,034.034 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6969/60103 objects degraded (11.595%); 285/20050 unfound (1.421%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4744330, u'degraded_objects': 6969, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83898667200, u'degraded_total': 60103, u'bytes_used': 68938797056, u'unfound_ratio': u'1.421', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 7590928, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 947, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.595', u'bytes_avail': 2266574065664, u'unfound_total': 20050}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:08,035.035 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:08,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1735 16 11612 11596 26.7304 0 - 2.36675 2014-04-01 16:10:08,793.793 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:09,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1736 16 11613 11597 26.7173 1.33333 4.5834 2.36694 2014-04-01 16:10:10,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1737 16 11617 11601 26.7112 16 0.188572 2.37175 2014-04-01 16:10:11,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1738 16 11619 11603 26.7004 8 0.108395 2.37136 2014-04-01 16:10:12,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1739 16 11619 11603 26.685 0 - 2.37136 2014-04-01 16:10:12,742.742 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:13,525.525 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:13,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:10:13.554552min lat: 0.063783 max lat: 348.961 avg lat: 2.37136 2014-04-01 16:10:13,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:10:13,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1740 16 11619 11603 26.6697 0 - 2.37136 2014-04-01 16:10:14,452.452 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:14,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1741 16 11619 11603 26.6544 0 - 2.37136 2014-04-01 16:10:15,178.178 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6896/60133 objects degraded (11.468%); 285/20060 unfound (1.421%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10058364, u'degraded_objects': 6896, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83940610240, u'degraded_total': 60133, u'bytes_used': 68772077568, u'unfound_ratio': u'1.421', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 5029182, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 951, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.468', u'bytes_avail': 2266740785152, u'unfound_total': 20060}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:15,179.179 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:15,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1742 16 11619 11603 26.6391 0 - 2.37136 2014-04-01 16:10:15,935.935 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:16,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1743 16 11622 11606 26.6307 2.4 0.15974 2.37619 2014-04-01 16:10:17,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1744 16 11622 11606 26.6154 0 - 2.37619 2014-04-01 16:10:18,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1745 16 11622 11606 26.6002 0 - 2.37619 2014-04-01 16:10:19,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1746 16 11624 11608 26.5895 2.66667 0.368737 2.37789 2014-04-01 16:10:19,860.860 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:20,689.689 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:20,848.848 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1747 16 11630 11614 26.588 24 37.7736 2.38951 2014-04-01 16:10:21,471.471 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:21,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1748 16 11639 11623 26.5934 36 0.186148 2.39448 2014-04-01 16:10:22,174.174 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6880/60151 objects degraded (11.438%); 285/20066 unfound (1.420%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4608509, u'degraded_objects': 6880, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83965776064, u'degraded_total': 60151, u'bytes_used': 68832428032, u'unfound_ratio': u'1.420', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 2765105, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 954, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.438', u'bytes_avail': 2266680434688, u'unfound_total': 20066}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:22,175.175 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:22,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1749 16 11643 11627 26.5873 16 0.15699 2.39375 2014-04-01 16:10:22,897.897 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:23,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1750 16 11643 11627 26.5722 0 - 2.39375 2014-04-01 16:10:24,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1751 16 11643 11627 26.557 0 - 2.39375 2014-04-01 16:10:25,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1752 16 11647 11631 26.551 5.33333 0.465956 2.39706 2014-04-01 16:10:26,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1753 16 11650 11634 26.5427 12 29.8353 2.39904 2014-04-01 16:10:26,662.662 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:27,545.545 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:27,705.705 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1754 16 11650 11634 26.5275 0 - 2.39904 2014-04-01 16:10:28,324.324 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:28,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1755 16 11650 11634 26.5124 0 - 2.39904 2014-04-01 16:10:29,077.077 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:09:33.245823', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871136, u'health': u'HEALTH_OK', u'kb_used': 10457848, u'store_stats': {u'bytes_total': 7352108, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:09:33.962873', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312116, u'health': u'HEALTH_OK', u'kb_used': 11016868, u'store_stats': {u'bytes_total': 8400599, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:09:33.495002', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437871052, u'health': u'HEALTH_OK', u'kb_used': 10457932, u'store_stats': {u'bytes_total': 8400492, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6847/60217 objects degraded (11.371%); 285/20088 unfound (1.419%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8631301, u'degraded_objects': 6847, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84058050752, u'degraded_total': 60217, u'bytes_used': 68962496512, u'unfound_ratio': u'1.419', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 14385501, u'op_per_sec': 3, u'unfound_objects': 285, u'version': 957, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.371', u'bytes_avail': 2266550366208, u'unfound_total': 20088}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:29,078.078 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:29,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1756 16 11663 11647 26.5269 17.3333 0.122888 2.40381 2014-04-01 16:10:29,905.905 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:30,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1757 16 11672 11656 26.5323 36 0.253345 2.40785 2014-04-01 16:10:31,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1758 16 11673 11657 26.5195 4 0.106934 2.40765 2014-04-01 16:10:33,140.140 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1759 16 11673 11657 26.5044 0 - 2.40765 2014-04-01 16:10:33,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:10:33.557379min lat: 0.063783 max lat: 348.961 avg lat: 2.40765 2014-04-01 16:10:33,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:10:33,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1760 16 11673 11657 26.4894 0 - 2.40765 2014-04-01 16:10:33,836.836 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:34,406.406 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:34,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1761 16 11673 11657 26.4743 0 - 2.40765 2014-04-01 16:10:35,343.343 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:35,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1762 16 11673 11657 26.4593 0 - 2.40765 2014-04-01 16:10:36,140.140 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6853/60283 objects degraded (11.368%); 285/20110 unfound (1.417%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12821670, u'degraded_objects': 6853, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84150325440, u'degraded_total': 60283, u'bytes_used': 69167104000, u'unfound_ratio': u'1.417', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 12821670, u'op_per_sec': 3, u'unfound_objects': 285, u'version': 961, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.368', u'bytes_avail': 2266345758720, u'unfound_total': 20110}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:36,141.141 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:36,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1763 16 11673 11657 26.4443 0 - 2.40765 2014-04-01 16:10:36,913.913 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:37,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1764 16 11674 11658 26.4316 0.666667 17.4035 2.40894 2014-04-01 16:10:38,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1765 16 11678 11662 26.4256 16 0.126949 2.40818 2014-04-01 16:10:39,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1766 16 11678 11662 26.4107 0 - 2.40818 2014-04-01 16:10:40,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1767 16 11685 11669 26.4116 14 0.111674 2.41038 2014-04-01 16:10:40,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:41,458.458 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:41,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1768 16 11685 11669 26.3966 0 - 2.41038 2014-04-01 16:10:42,252.252 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:42,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1769 16 11688 11672 26.3885 6 0.15854 2.41344 2014-04-01 16:10:43,086.086 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.013570', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.036897', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6808/60313 objects degraded (11.288%); 285/20120 unfound (1.417%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3825095, u'degraded_objects': 6808, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84192268480, u'degraded_total': 60313, u'bytes_used': 69260660736, u'unfound_ratio': u'1.417', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 7650190, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 965, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.288', u'bytes_avail': 2266252201984, u'unfound_total': 20120}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:43,087.087 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:43,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1770 16 11688 11672 26.3736 0 - 2.41344 2014-04-01 16:10:43,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:44,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1771 16 11688 11672 26.3587 0 - 2.41344 2014-04-01 16:10:45,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1772 16 11688 11672 26.3438 0 - 2.41344 2014-04-01 16:10:46,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1773 16 11688 11672 26.329 0 - 2.41344 2014-04-01 16:10:47,558.558 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:47,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1774 16 11689 11673 26.3164 0.8 16.8381 2.41468 2014-04-01 16:10:48,356.356 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:48,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1775 16 11692 11676 26.3083 12 0.072546 2.41562 2014-04-01 16:10:49,330.330 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:49,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1776 16 11692 11676 26.2935 0 - 2.41562 2014-04-01 16:10:50,049.049 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6774/60328 objects degraded (11.229%); 285/20125 unfound (1.416%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12539025, u'degraded_objects': 6774, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84213240000, u'degraded_total': 60328, u'bytes_used': 69433905152, u'unfound_ratio': u'1.416', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 2507805, u'op_per_sec': 0, u'unfound_objects': 285, u'version': 969, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.229', u'bytes_avail': 2266078957568, u'unfound_total': 20125}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:50,050.050 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:50,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1777 16 11692 11676 26.2787 0 - 2.41562 2014-04-01 16:10:50,828.828 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:51,750.750 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1778 16 11692 11676 26.2639 0 - 2.41562 2014-04-01 16:10:52,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1779 16 11697 11681 26.2604 5 12.1366 2.41903 2014-04-01 16:10:53,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:10:53.560204min lat: 0.063783 max lat: 348.961 avg lat: 2.41952 2014-04-01 16:10:53,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:10:53,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1780 16 11699 11683 26.2501 8 0.244308 2.41952 2014-04-01 16:10:54,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1781 16 11700 11684 26.2376 4 25.5054 2.42149 2014-04-01 16:10:54,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:55,498.498 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:55,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1782 16 11702 11686 26.2274 8 26.496 2.42441 2014-04-01 16:10:57,041.041 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1783 16 11707 11691 26.2239 20 0.167556 2.42657 2014-04-01 16:10:57,372.372 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:10:57,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1784 16 11708 11692 26.2115 4 8.88569 2.42712 2014-04-01 16:10:58,088.088 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6678/60373 objects degraded (11.061%); 285/20140 unfound (1.415%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 6678, u'num_pgs': 168, u'data_bytes': 84276154560, u'degraded_total': 60373, u'bytes_used': 69191753728, u'unfound_ratio': u'1.415', u'op_per_sec': 3, u'write_bytes_sec': 12889650, u'unfound_objects': 285, u'version': 974, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.061', u'bytes_avail': 2266321108992, u'unfound_total': 20140}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:10:58,089.089 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:10:58,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1785 16 11711 11695 26.2035 12 0.167532 2.42945 2014-04-01 16:10:58,936.936 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:10:59,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1786 16 11711 11695 26.1888 0 - 2.42945 2014-04-01 16:11:00,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1787 16 11715 11699 26.1831 8 0.505121 2.43383 2014-04-01 16:11:01,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1788 16 11715 11699 26.1685 0 - 2.43383 2014-04-01 16:11:02,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1789 16 11715 11699 26.1538 0 - 2.43383 2014-04-01 16:11:02,887.887 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:03,610.610 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:03,766.766 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1790 16 11715 11699 26.1392 0 - 2.43383 2014-04-01 16:11:04,548.548 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:04,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1791 16 11715 11699 26.1246 0 - 2.43383 2014-04-01 16:11:05,322.322 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6665/60415 objects degraded (11.032%); 285/20154 unfound (1.414%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1889377, u'degraded_objects': 6665, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84334874816, u'degraded_total': 60415, u'bytes_used': 69403799552, u'unfound_ratio': u'1.414', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 9446888, u'op_per_sec': 2, u'unfound_objects': 285, u'version': 977, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.032', u'bytes_avail': 2266109063168, u'unfound_total': 20154}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:05,323.323 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:05,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1792 16 11720 11704 26.1212 4 0.137225 2.43658 2014-04-01 16:11:06,139.139 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:06,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1793 16 11727 11711 26.1223 28 0.418094 2.44208 2014-04-01 16:11:07,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1794 16 11735 11719 26.1255 32 15.1249 2.44484 2014-04-01 16:11:08,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1795 16 11745 11729 26.1333 40 0.384141 2.44353 2014-04-01 16:11:09,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1796 16 11747 11731 26.1232 8 12.8498 2.4453 2014-04-01 16:11:10,073.073 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:10,782.782 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:10,945.945 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1797 16 11750 11734 26.1153 12 12.0173 2.44574 2014-04-01 16:11:11,570.570 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:11,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1798 16 11750 11734 26.1008 0 - 2.44574 2014-04-01 16:11:12,283.283 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6637/60487 objects degraded (10.973%); 285/20178 unfound (1.412%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13891949, u'degraded_objects': 6637, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84435538112, u'degraded_total': 60487, u'bytes_used': 69555990528, u'unfound_ratio': u'1.412', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 20206471, u'op_per_sec': 4, u'unfound_objects': 285, u'version': 981, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.973', u'bytes_avail': 2265956872192, u'unfound_total': 20178}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:12,284.284 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:12,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1799 16 11750 11734 26.0863 0 - 2.44574 2014-04-01 16:11:13,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:13,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 07:11:13.563091min lat: 0.063783 max lat: 348.961 avg lat: 2.44574 2014-04-01 16:11:13,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 16:11:13,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1800 16 11750 11734 26.0718 0 - 2.44574 2014-04-01 16:11:14,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1801 16 11750 11734 26.0573 0 - 2.44574 2014-04-01 16:11:15,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1802 16 11751 11735 26.0451 0.8 7.89625 2.44621 2014-04-01 16:11:16,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1803 16 11751 11735 26.0306 0 - 2.44621 2014-04-01 16:11:16,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:17,609.609 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:17,767.767 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1804 14 11751 11737 26.0206 4 12.0816 2.44759 2014-04-01 16:11:18,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:18,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1805 14 11751 11737 26.0062 0 - 2.44759 2014-04-01 16:11:19,118.118 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6613/60535 objects degraded (10.924%); 285/20194 unfound (1.411%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2879829, u'degraded_objects': 6613, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84502646976, u'degraded_total': 60535, u'bytes_used': 69745303552, u'unfound_ratio': u'1.411', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 7679544, u'op_per_sec': 1, u'unfound_objects': 285, u'version': 984, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.924', u'bytes_avail': 2265767559168, u'unfound_total': 20194}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:19,119.119 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:19,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1806 14 11751 11737 25.9918 0 - 2.44759 2014-04-01 16:11:19,840.840 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Total time run: 1806.924917 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Total writes made: 11751 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Write size: 4194304 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Bandwidth (MB/sec): 26.013 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Stddev Bandwidth: 33.5091 2014-04-01 16:11:20,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Max bandwidth (MB/sec): 148 2014-04-01 16:11:20,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Min bandwidth (MB/sec): 0 2014-04-01 16:11:20,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Average Latency: 2.45979 2014-04-01 16:11:20,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Stddev Latency: 6.3515 2014-04-01 16:11:20,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Max latency: 348.961 2014-04-01 16:11:20,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Min latency: 0.063783 2014-04-01 16:11:23,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:24,390.390 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:25,203.203 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:25,961.961 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:10:33.246081', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837844, u'health': u'HEALTH_OK', u'kb_used': 10491140, u'store_stats': {u'bytes_total': 8400684, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157788}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6480/60517 objects degraded (10.708%); 285/20188 unfound (1.412%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6287425, u'degraded_objects': 6480, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84473286860, u'degraded_total': 60517, u'bytes_used': 69882458112, u'unfound_ratio': u'1.412', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 2514970, u'op_per_sec': 4, u'unfound_objects': 285, u'version': 988, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.708', u'bytes_avail': 2265630404608, u'unfound_total': 20188}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:25,962.962 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:26,725.725 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:30,528.528 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:31,196.196 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:32,703.703 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:33,615.615 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:10:33.963606', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437265824, u'health': u'HEALTH_OK', u'kb_used': 11063160, u'store_stats': {u'bytes_total': 10497751, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157703}}, {u'last_updated': u'2014-04-01 07:10:33.495270', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437837660, u'health': u'HEALTH_OK', u'kb_used': 10491324, u'store_stats': {u'bytes_total': 10497644, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3157596}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6392/60391 objects degraded (10.584%); 285/20146 unfound (1.415%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 6392, u'num_pgs': 168, u'data_bytes': 84292931788, u'degraded_total': 60391, u'bytes_used': 69961949184, u'unfound_ratio': u'1.415', u'op_per_sec': 1, u'write_bytes_sec': 0, u'unfound_objects': 285, u'version': 992, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.584', u'bytes_avail': 2265550913536, u'unfound_total': 20146}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:33,616.616 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:34,483.483 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:38,585.585 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:39,361.361 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:40,309.309 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:41,188.188 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6316/60274 objects degraded (10.479%); 285/20107 unfound (1.417%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12525874, u'degraded_objects': 6316, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 84129353932, u'degraded_total': 60274, u'bytes_used': 70064316416, u'unfound_ratio': u'1.417', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'unfound_objects': 285, u'version': 996, u'pgs_by_state': [{u'count': 9, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.479', u'bytes_avail': 2265448546304, u'unfound_total': 20107}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:41,189.189 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:41,949.949 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:46,031.031 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:46,858.858 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:48,026.026 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:48,997.997 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6226/60160 objects degraded (10.349%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3821565, u'degraded_objects': 6226, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83969970380, u'degraded_total': 60160, u'bytes_used': 70107701248, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1000, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.349', u'bytes_avail': 2265405161472}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:48,998.998 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:49,955.955 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:54,252.252 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:55,040.040 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:11:55,971.971 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:11:56,699.699 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6118/60028 objects degraded (10.192%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6534714, u'degraded_objects': 6118, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83785421004, u'degraded_total': 60028, u'bytes_used': 70207328256, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1004, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.192', u'bytes_avail': 2265305534464}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:11:56,700.700 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:11:57,478.478 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:01,243.243 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:01,956.956 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:02,739.739 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:03,470.470 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6060/59965 objects degraded (10.106%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4787157, u'degraded_objects': 6060, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83697340620, u'degraded_total': 59965, u'bytes_used': 70262751232, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1007, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.106', u'bytes_avail': 2265250111488}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:03,471.471 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:04,342.342 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:08,265.265 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:09,267.267 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:10,079.079 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:10,849.849 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5992/59920 objects degraded (10.000%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10025199, u'degraded_objects': 5992, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83634426060, u'degraded_total': 59920, u'bytes_used': 70880567296, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1011, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.000', u'bytes_avail': 2264632295424}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:10,850.850 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:11,553.553 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:15,317.317 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:15,940.940 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:16,969.969 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:17,935.935 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5899/59839 objects degraded (9.858%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 5899, u'num_pgs': 168, u'data_bytes': 83521179852, u'degraded_total': 59839, u'bytes_used': 70953893888, u'op_per_sec': 4, u'write_bytes_sec': 0, u'version': 1015, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.858', u'bytes_avail': 2264558968832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:17,937.937 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:18,713.713 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:22,650.650 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:23,550.550 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:24,482.482 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:25,223.223 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5798/59752 objects degraded (9.703%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6516790, u'degraded_objects': 5798, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83399545036, u'degraded_total': 59752, u'bytes_used': 71072776192, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1018, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 145, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.703', u'bytes_avail': 2264440086528}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:25,224.224 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:25,957.957 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:29,919.919 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:31,502.502 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:32,292.292 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:33,073.073 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:11:33.246337', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816348, u'health': u'HEALTH_OK', u'kb_used': 10512636, u'store_stats': {u'bytes_total': 4526835, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478243}}, {u'last_updated': u'2014-04-01 07:11:33.963847', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437227160, u'health': u'HEALTH_OK', u'kb_used': 11101824, u'store_stats': {u'bytes_total': 4527006, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478414}}, {u'last_updated': u'2014-04-01 07:11:33.495581', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437816264, u'health': u'HEALTH_OK', u'kb_used': 10512720, u'store_stats': {u'bytes_total': 4526928, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3478336}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5692/59614 objects degraded (9.548%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8596703, u'degraded_objects': 5692, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 83206607052, u'degraded_total': 59614, u'bytes_used': 71104491520, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1022, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 145, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.548', u'bytes_avail': 2264408371200}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:33,074.074 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:33,776.776 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:37,570.570 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:38,305.305 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:39,133.133 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:39,969.969 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5583/59476 objects degraded (9.387%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 5583, u'num_pgs': 168, u'data_bytes': 83013669068, u'degraded_total': 59476, u'bytes_used': 71026663424, u'op_per_sec': 11, u'write_bytes_sec': 0, u'version': 1025, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.387', u'bytes_avail': 2264486199296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:39,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:40,787.787 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:44,580.580 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:45,296.296 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:46,072.072 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:46,878.878 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5459/59344 objects degraded (9.199%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4681728, u'degraded_objects': 5459, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82829119692, u'degraded_total': 59344, u'bytes_used': 70858428416, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1030, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.199', u'bytes_avail': 2264654434304}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:46,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:50,724.724 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:54,505.505 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:55,224.224 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:12:56,186.186 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:12:57,057.057 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5337/59158 objects degraded (9.022%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7600810, u'degraded_objects': 5337, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82569072844, u'degraded_total': 59158, u'bytes_used': 70960037888, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1036, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.022', u'bytes_avail': 2264552824832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:12:57,058.058 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:12:57,822.822 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:01,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:02,319.319 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:03,399.399 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:04,284.284 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5279/59059 objects degraded (8.939%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 5279, u'num_pgs': 168, u'data_bytes': 82430660812, u'degraded_total': 59059, u'bytes_used': 71005036544, u'op_per_sec': 6, u'write_bytes_sec': 0, u'version': 1039, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.939', u'bytes_avail': 2264507826176}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:04,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:05,357.357 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:09,121.121 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:10,009.009 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:10,802.802 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:11,587.587 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5202/58888 objects degraded (8.834%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3691721, u'degraded_objects': 5202, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 82191585484, u'degraded_total': 58888, u'bytes_used': 71073071104, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1043, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.834', u'bytes_avail': 2264439791616}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:11,588.588 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:12,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:16,141.141 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:16,927.927 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:17,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:18,764.764 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5100/58717 objects degraded (8.686%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 5100, u'num_pgs': 168, u'data_bytes': 81952510156, u'degraded_total': 58717, u'bytes_used': 71074107392, u'op_per_sec': 11, u'write_bytes_sec': 0, u'version': 1048, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.686', u'bytes_avail': 2264438755328}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:18,766.766 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:19,518.518 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:23,303.303 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:24,014.014 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:24,798.798 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:25,524.524 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5038/58600 objects degraded (8.597%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6240437, u'degraded_objects': 5038, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81788932300, u'degraded_total': 58600, u'bytes_used': 71182004224, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1052, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.597', u'bytes_avail': 2264330858496}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:25,525.525 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:26,440.440 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:30,373.373 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:31,183.183 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:32,413.413 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:33,267.267 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:12:33.246588', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782940, u'health': u'HEALTH_OK', u'kb_used': 10546044, u'store_stats': {u'bytes_total': 5761229, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:12:33.964104', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437176404, u'health': u'HEALTH_OK', u'kb_used': 11152580, u'store_stats': {u'bytes_total': 5761411, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:12:33.495823', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437782740, u'health': u'HEALTH_OK', u'kb_used': 10546244, u'store_stats': {u'bytes_total': 5761295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4935/58393 objects degraded (8.451%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9347100, u'degraded_objects': 4935, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81499525324, u'degraded_total': 58393, u'bytes_used': 71221395456, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1056, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.451', u'bytes_avail': 2264291467264}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:33,268.268 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:34,087.087 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:38,148.148 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:38,862.862 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:39,650.650 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:40,356.356 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4855/58255 objects degraded (8.334%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8746116, u'degraded_objects': 4855, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81306587340, u'degraded_total': 58255, u'bytes_used': 71257333760, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1060, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 146, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.334', u'bytes_avail': 2264255528960}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:40,358.358 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:41,259.259 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:45,037.037 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:45,696.696 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:46,636.636 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:47,358.358 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4739/58150 objects degraded (8.150%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10609495, u'degraded_objects': 4739, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81159786700, u'degraded_total': 58150, u'bytes_used': 71326695424, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1064, u'pgs_by_state': [{u'count': 7, u'state_name': u'active+recovery_wait'}, {u'count': 147, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.150', u'bytes_avail': 2264186167296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:47,359.359 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:48,094.094 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:51,860.860 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:52,633.633 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:53,421.421 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:13:54,213.213 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4671/58036 objects degraded (8.048%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1291255, u'degraded_objects': 4671, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 81000403148, u'degraded_total': 58036, u'bytes_used': 71355301888, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1068, u'pgs_by_state': [{u'count': 7, u'state_name': u'active+recovery_wait'}, {u'count': 148, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.048', u'bytes_avail': 2264157560832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:13:54,214.214 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:55,159.159 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:13:58,934.934 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:13:59,651.651 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:00,590.590 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:01,374.374 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4616/57937 objects degraded (7.967%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7336222, u'degraded_objects': 4616, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80861991116, u'degraded_total': 57937, u'bytes_used': 71443017728, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1072, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 148, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.967', u'bytes_avail': 2264069844992}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:01,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:02,098.098 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:05,875.875 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:06,456.456 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:07,395.395 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:08,103.103 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4545/57865 objects degraded (7.854%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1826274, u'degraded_objects': 4545, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80761327820, u'degraded_total': 57865, u'bytes_used': 71988617216, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1077, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 148, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.854', u'bytes_avail': 2263524245504}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:08,105.105 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:09,881.881 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:13,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:14,550.550 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:15,482.482 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:16,360.360 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4490/57790 objects degraded (7.770%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5023986, u'degraded_objects': 4490, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80656470220, u'degraded_total': 57790, u'bytes_used': 72066461696, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1081, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 148, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.770', u'bytes_avail': 2263446401024}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:16,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:17,144.144 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:21,108.108 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:21,671.671 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:22,621.621 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:23,386.386 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4389/57637 objects degraded (7.615%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4769663, u'degraded_objects': 4389, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80442560716, u'degraded_total': 57637, u'bytes_used': 72093941760, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1085, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.615', u'bytes_avail': 2263418920960}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:23,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:24,140.140 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:27,916.916 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:28,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:29,484.484 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:30,186.186 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:13:33.246861', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437744044, u'health': u'HEALTH_OK', u'kb_used': 10584940, u'store_stats': {u'bytes_total': 6809805, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664061}}, {u'last_updated': u'2014-04-01 07:13:33.964360', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437127280, u'health': u'HEALTH_OK', u'kb_used': 11201704, u'store_stats': {u'bytes_total': 8907139, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664243}}, {u'last_updated': u'2014-04-01 07:13:33.496192', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437743868, u'health': u'HEALTH_OK', u'kb_used': 10585116, u'store_stats': {u'bytes_total': 8907023, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4302/57532 objects degraded (7.478%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8751674, u'degraded_objects': 4302, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80295760076, u'degraded_total': 57532, u'bytes_used': 72146247680, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1089, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.478', u'bytes_avail': 2263366615040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:30,187.187 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:30,886.886 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:34,817.817 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:35,545.545 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:36,339.339 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:37,074.074 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4239/57397 objects degraded (7.385%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10061906, u'degraded_objects': 4239, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 80107016396, u'degraded_total': 57397, u'bytes_used': 72194486272, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1092, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.385', u'bytes_avail': 2263318376448}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:37,075.075 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:38,047.047 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:41,830.830 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:42,602.602 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:43,706.706 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:44,497.497 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4152/57283 objects degraded (7.248%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5332029, u'degraded_objects': 4152, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79947632844, u'degraded_total': 57283, u'bytes_used': 72271675392, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1096, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.248', u'bytes_avail': 2263241187328}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:44,499.499 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:45,213.213 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:49,151.151 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:49,876.876 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:50,822.822 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:51,554.554 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4102/57202 objects degraded (7.171%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12182854, u'degraded_objects': 4102, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79834386636, u'degraded_total': 57202, u'bytes_used': 72320724992, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1101, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.171', u'bytes_avail': 2263192137728}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:51,555.555 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:52,259.259 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:56,189.189 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:56,816.816 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:14:57,759.759 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:14:58,555.555 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4024/57025 objects degraded (7.057%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 4024, u'num_pgs': 168, u'data_bytes': 79586922700, u'degraded_total': 57025, u'bytes_used': 72355483648, u'op_per_sec': 12, u'write_bytes_sec': 0, u'version': 1106, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.057', u'bytes_avail': 2263157379072}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:14:58,556.556 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:14:59,307.307 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:03,088.088 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:03,805.805 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:04,744.744 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:05,503.503 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3957/56965 objects degraded (6.946%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13633942, u'degraded_objects': 3957, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79503036620, u'degraded_total': 56965, u'bytes_used': 72455593984, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1110, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.946', u'bytes_avail': 2263057268736}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:05,504.504 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:06,421.421 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:10,347.347 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:11,183.183 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:11,991.991 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:12,717.717 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3871/56743 objects degraded (6.822%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6691643, u'degraded_objects': 3871, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79192658124, u'degraded_total': 56743, u'bytes_used': 72495607808, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1114, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.822', u'bytes_avail': 2263017254912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:12,719.719 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:13,505.505 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:17,317.317 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:17,650.650 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:15:17.574039 7f82984e3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/7690 pipe(0x7f82a001a8e0 sd=12 :52836 s=2 pgs=4 cs=1 l=1 c=0x7f82a0018200).injecting socket failure 2014-04-01 16:15:18,047.047 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:18,864.864 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:19,473.473 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:15:19.397007 7f098f7fe700 0 monclient: hunting for new mon 2014-04-01 16:15:19,591.591 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3827/56650 objects degraded (6.756%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1286492, u'degraded_objects': 3827, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 79062634700, u'degraded_total': 56650, u'bytes_used': 72504352768, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1117, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.756', u'bytes_avail': 2263008509952}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:19,592.592 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:20,306.306 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:24,227.227 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:24,941.941 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:25,765.765 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:26,469.469 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:14:33.247198', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715412, u'health': u'HEALTH_OK', u'kb_used': 10613572, u'store_stats': {u'bytes_total': 5505274, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3408106}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:14:33.496506', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437715348, u'health': u'HEALTH_OK', u'kb_used': 10613636, u'store_stats': {u'bytes_total': 6553736, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407992}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3750/56518 objects degraded (6.635%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6542073, u'degraded_objects': 3750, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78878085324, u'degraded_total': 56518, u'bytes_used': 72605736960, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1120, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.635', u'bytes_avail': 2262907125760}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:26,470.470 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:27,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:30,944.944 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:31,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:33,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:33,844.844 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:14:33.964608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437093956, u'health': u'HEALTH_OK', u'kb_used': 11235028, u'store_stats': {u'bytes_total': 6553683, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3407939}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3669/56350 objects degraded (6.511%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12423996, u'degraded_objects': 3669, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78643204300, u'degraded_total': 56350, u'bytes_used': 72632549376, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1124, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.511', u'bytes_avail': 2262880313344}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:33,846.846 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:34,662.662 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:38,545.545 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:39,134.134 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:40,071.071 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:40,839.839 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.011560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.029653', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3588/56236 objects degraded (6.380%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11289061, u'degraded_objects': 3588, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78483820748, u'degraded_total': 56236, u'bytes_used': 72707469312, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1128, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.380', u'bytes_avail': 2262805393408}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:40,840.840 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:41,542.542 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:45,312.312 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:46,149.149 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:46,934.934 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:47,705.705 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3507/56086 objects degraded (6.253%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4785329, u'degraded_objects': 3507, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 78274105548, u'degraded_total': 56086, u'bytes_used': 72722972672, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1132, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 150, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.253', u'bytes_avail': 2262789890048}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:47,706.706 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:48,404.404 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:52,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:53,163.163 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:53,967.967 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:15:54,747.747 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3393/55879 objects degraded (6.072%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10778662, u'degraded_objects': 3393, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 77984698572, u'degraded_total': 55879, u'bytes_used': 72510537728, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1137, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.072', u'bytes_avail': 2263002324992}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:15:54,748.748 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:15:55,770.770 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:15:59,549.549 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:00,260.260 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:01,036.036 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:01,767.767 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3329/55801 objects degraded (5.966%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7557447, u'degraded_objects': 3329, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 77875646668, u'degraded_total': 55801, u'bytes_used': 72553951232, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1140, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.966', u'bytes_avail': 2262958911488}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:01,768.768 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:02,552.552 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:06,512.512 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:07,149.149 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:08,090.090 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:08,854.854 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3244/55648 objects degraded (5.829%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5147244, u'degraded_objects': 3244, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 77661737164, u'degraded_total': 55648, u'bytes_used': 72684605440, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1144, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.829', u'bytes_avail': 2262828257280}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:08,855.855 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:09,553.553 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:13,321.321 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:17,043.043 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:17,833.833 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:18,617.617 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3133/55471 objects degraded (5.648%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12576573, u'degraded_objects': 3133, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 77414273228, u'degraded_total': 55471, u'bytes_used': 72741269504, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1148, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.648', u'bytes_avail': 2262771593216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:18,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:19,316.316 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:23,235.235 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:23,952.952 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:24,734.734 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:25,528.528 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3043/55339 objects degraded (5.499%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12410417, u'degraded_objects': 3043, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 77229723852, u'degraded_total': 55339, u'bytes_used': 72832380928, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1152, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.499', u'bytes_avail': 2262680481792}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:25,529.529 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:26,309.309 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:30,072.072 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:30,802.802 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:31,596.596 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:32,306.306 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:15:33.247483', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437684068, u'health': u'HEALTH_OK', u'kb_used': 10644916, u'store_stats': {u'bytes_total': 4141556, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:15:33.964854', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437062356, u'health': u'HEALTH_OK', u'kb_used': 11266628, u'store_stats': {u'bytes_total': 5190032, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:15:33.496817', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437683988, u'health': u'HEALTH_OK', u'kb_used': 10644996, u'store_stats': {u'bytes_total': 5189979, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2993/55168 objects degraded (5.425%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8416830, u'degraded_objects': 2993, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76990648524, u'degraded_total': 55168, u'bytes_used': 72904425472, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1155, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.425', u'bytes_avail': 2262608437248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:32,307.307 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:33,025.025 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:36,789.789 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:37,522.522 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:38,306.306 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:39,032.032 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2934/55066 objects degraded (5.328%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12468851, u'degraded_objects': 2934, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76848042188, u'degraded_total': 55066, u'bytes_used': 72941019136, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1158, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.328', u'bytes_avail': 2262571843584}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:39,033.033 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:39,741.741 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:43,509.509 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:44,301.301 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:45,096.096 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:46,728.728 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2852/54937 objects degraded (5.191%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10085501, u'degraded_objects': 2852, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76667687116, u'degraded_total': 54937, u'bytes_used': 73001222144, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1162, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.191', u'bytes_avail': 2262511640576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:46,729.729 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:47,440.440 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:51,373.373 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:51,983.983 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:52,923.923 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:16:53,644.644 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2763/54829 objects degraded (5.039%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8872781, u'degraded_objects': 2763, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76516692172, u'degraded_total': 54829, u'bytes_used': 73051041792, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1167, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.039', u'bytes_avail': 2262461820928}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:16:53,645.645 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:54,347.347 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:58,148.148 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:16:58,970.970 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:16:59,760.760 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:00,470.470 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2689/54736 objects degraded (4.913%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7389413, u'degraded_objects': 2689, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76386668748, u'degraded_total': 54736, u'bytes_used': 73168617472, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1171, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.913', u'bytes_avail': 2262344245248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:00,470.470 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:01,233.233 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:04,995.995 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:06,018.018 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:06,951.951 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:07,694.694 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2602/54580 objects degraded (4.767%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5582837, u'degraded_objects': 2602, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76168564940, u'degraded_total': 54580, u'bytes_used': 73198850048, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1176, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.767', u'bytes_avail': 2262314012672}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:07,695.695 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:08,465.465 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:12,409.409 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:13,216.216 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:14,185.185 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:15,003.003 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2527/54475 objects degraded (4.639%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13879654, u'degraded_objects': 2527, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 76021764300, u'degraded_total': 54475, u'bytes_used': 73290768384, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1180, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.639', u'bytes_avail': 2262222094336}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:15,004.004 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:15,766.766 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:19,531.531 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:20,253.253 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:21,182.182 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:21,918.918 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2481/54340 objects degraded (4.566%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5545760, u'degraded_objects': 2481, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75833020620, u'degraded_total': 54340, u'bytes_used': 73327947776, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1183, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.566', u'bytes_avail': 2262184914944}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:21,919.919 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:22,736.736 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:26,655.655 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:27,373.373 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:28,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:29,108.108 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:16:33.247746', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643652, u'health': u'HEALTH_OK', u'kb_used': 10685332, u'store_stats': {u'bytes_total': 6238708, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:16:33.965155', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437015216, u'health': u'HEALTH_OK', u'kb_used': 11313768, u'store_stats': {u'bytes_total': 7287184, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:16:33.497127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437643608, u'health': u'HEALTH_OK', u'kb_used': 10685376, u'store_stats': {u'bytes_total': 7287131, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2409/54223 objects degraded (4.443%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8611472, u'degraded_objects': 2409, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75669442764, u'degraded_total': 54223, u'bytes_used': 73377259520, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1187, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.443', u'bytes_avail': 2262135603200}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:29,109.109 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:29,862.862 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:33,992.992 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:34,833.833 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:35,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:36,436.436 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2337/54160 objects degraded (4.315%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14018999, u'degraded_objects': 2337, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75581362380, u'degraded_total': 54160, u'bytes_used': 73475473408, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1191, u'pgs_by_state': [{u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.315', u'bytes_avail': 2262037389312}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:36,438.438 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:37,151.151 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:41,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:41,847.847 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:42,795.795 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:43,507.507 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2272/54100 objects degraded (4.200%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12492717, u'degraded_objects': 2272, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75497476300, u'degraded_total': 54100, u'bytes_used': 73535516672, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 1195, u'pgs_by_state': [{u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.200', u'bytes_avail': 2261977346048}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:43,508.508 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:44,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:48,150.150 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:48,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:49,851.851 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:50,587.587 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2218/54064 objects degraded (4.103%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13195398, u'degraded_objects': 2218, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75447144652, u'degraded_total': 54064, u'bytes_used': 73629847552, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1199, u'pgs_by_state': [{u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.103', u'bytes_avail': 2261883015168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:50,588.588 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:51,297.297 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:55,219.219 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:55,940.940 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:17:56,726.726 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:17:57,490.490 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2143/54052 objects degraded (3.965%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12421721, u'degraded_objects': 2143, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75430367436, u'degraded_total': 54052, u'bytes_used': 73706737664, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 1203, u'pgs_by_state': [{u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.965', u'bytes_avail': 2261806125056}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:17:57,492.492 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:17:58,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:02,163.163 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:02,995.995 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:03,959.959 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:04,702.702 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1758/54040 objects degraded (3.253%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9039398, u'degraded_objects': 1758, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75413590220, u'degraded_total': 54040, u'bytes_used': 73775595520, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1206, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.253', u'bytes_avail': 2261737267200}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:04,703.703 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:05,403.403 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:09,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:10,061.061 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:10,838.838 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:11,553.553 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1713/54016 objects degraded (3.171%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14827935, u'degraded_objects': 1713, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75380035788, u'degraded_total': 54016, u'bytes_used': 73467654144, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1209, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.171', u'bytes_avail': 2262045208576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:11,554.554 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:12,260.260 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:16,021.021 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:16,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:17,688.688 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:18,464.464 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1647/53986 objects degraded (3.051%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10277848, u'degraded_objects': 1647, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75338092748, u'degraded_total': 53986, u'bytes_used': 73533480960, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1214, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.051', u'bytes_avail': 2261979381760}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:18,465.465 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:19,169.169 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:22,937.937 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:23,653.653 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:24,434.434 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:25,144.144 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1593/53890 objects degraded (2.956%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12974512, u'degraded_objects': 1593, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75203875020, u'degraded_total': 53890, u'bytes_used': 73641447424, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1217, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.956', u'bytes_avail': 2261871415296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:25,145.145 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:25,851.851 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:29,620.620 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:30,335.335 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:31,117.117 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:31,887.887 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:17:33.247983', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602800, u'health': u'HEALTH_OK', u'kb_used': 10726184, u'store_stats': {u'bytes_total': 8335860, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:17:33.965465', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436966720, u'health': u'HEALTH_OK', u'kb_used': 11362264, u'store_stats': {u'bytes_total': 10432912, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:17:33.497348', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437602716, u'health': u'HEALTH_OK', u'kb_used': 10726268, u'store_stats': {u'bytes_total': 10432859, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1544/53839 objects degraded (2.868%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2502500, u'degraded_objects': 1544, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75132571852, u'degraded_total': 53839, u'bytes_used': 73654165504, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1220, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.868', u'bytes_avail': 2261858697216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:31,888.888 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:32,651.651 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:36,572.572 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:37,343.343 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:38,127.127 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:38,895.895 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1505/53761 objects degraded (2.799%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10227842, u'degraded_objects': 1505, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 75023519948, u'degraded_total': 53761, u'bytes_used': 73409466368, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1224, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.799', u'bytes_avail': 2262103396352}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:38,896.896 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:39,679.679 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:43,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:44,325.325 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:45,268.268 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:45,998.998 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1466/53701 objects degraded (2.730%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13951850, u'degraded_objects': 1466, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74939633868, u'degraded_total': 53701, u'bytes_used': 73527230464, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1227, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.730', u'bytes_avail': 2261985632256}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:45,999.999 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:46,857.857 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:50,776.776 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:51,494.494 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:52,429.429 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:53,147.147 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1401/53656 objects degraded (2.611%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14407963, u'degraded_objects': 1401, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74876719308, u'degraded_total': 53656, u'bytes_used': 73625329664, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1232, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.611', u'bytes_avail': 2261887533056}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:53,148.148 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:53,851.851 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:57,635.635 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:18:58,378.378 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:18:59,175.175 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:18:59,949.949 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1374/53626 objects degraded (2.562%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9018903, u'degraded_objects': 1374, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74834776268, u'degraded_total': 53626, u'bytes_used': 73689542656, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1235, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.562', u'bytes_avail': 2261823320064}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:18:59,950.950 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:00,708.708 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:04,645.645 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:05,417.417 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:06,205.205 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:06,976.976 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1303/53554 objects degraded (2.433%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11263653, u'degraded_objects': 1303, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74734112972, u'degraded_total': 53554, u'bytes_used': 73786970112, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1239, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.433', u'bytes_avail': 2261725892608}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:06,977.977 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:07,927.927 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:11,848.848 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:12,604.604 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:13,384.384 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:14,188.188 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1236/53458 objects degraded (2.312%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12449961, u'degraded_objects': 1236, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74599895244, u'degraded_total': 53458, u'bytes_used': 73860362240, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1242, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.312', u'bytes_avail': 2261652500480}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:14,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:14,893.893 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:18,991.991 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:19,775.775 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:20,712.712 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:21,478.478 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1183/53443 objects degraded (2.214%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11136164, u'degraded_objects': 1183, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74578923724, u'degraded_total': 53443, u'bytes_used': 73984065536, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1247, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.214', u'bytes_avail': 2261528797184}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:21,479.479 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:22,182.182 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:25,955.955 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:26,520.520 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:27,468.468 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:28,172.172 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:18:33.248237', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572076, u'health': u'HEALTH_OK', u'kb_used': 10756908, u'store_stats': {u'bytes_total': 9384436, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092964}}, {u'last_updated': u'2014-04-01 07:18:33.965757', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436925604, u'health': u'HEALTH_OK', u'kb_used': 11403380, u'store_stats': {u'bytes_total': 12530064, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092864}}, {u'last_updated': u'2014-04-01 07:18:33.497584', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437572044, u'health': u'HEALTH_OK', u'kb_used': 10756940, u'store_stats': {u'bytes_total': 12530011, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3092811}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1117/53413 objects degraded (2.091%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10434653, u'degraded_objects': 1117, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74536980684, u'degraded_total': 53413, u'bytes_used': 74030530560, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 1251, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.091', u'bytes_avail': 2261482332160}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:28,173.173 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:28,935.935 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:33,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:37,052.052 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:38,006.006 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:38,768.768 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1025/53284 objects degraded (1.924%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 15455445, u'degraded_objects': 1025, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74356625612, u'degraded_total': 53284, u'bytes_used': 74122174464, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1257, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.924', u'bytes_avail': 2261390688256}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:38,769.769 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:39,463.463 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:43,246.246 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:43,975.975 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:44,761.761 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:45,528.528 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 978/53266 objects degraded (1.836%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8998153, u'degraded_objects': 978, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74331459788, u'degraded_total': 53266, u'bytes_used': 74240323584, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 1260, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.836', u'bytes_avail': 2261272539136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:45,529.529 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:46,234.234 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:50,155.155 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:50,713.713 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:51,650.650 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:52,423.423 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 932/53125 objects degraded (1.754%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13818375, u'degraded_objects': 932, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 74134327500, u'degraded_total': 53125, u'bytes_used': 74315890688, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1264, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.754', u'bytes_avail': 2261196972032}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:52,424.424 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:53,134.134 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:57,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:19:57,787.787 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:19:58,575.575 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:19:59,365.365 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 877/53077 objects degraded (1.652%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 877, u'num_pgs': 168, u'data_bytes': 74067218636, u'degraded_total': 53077, u'bytes_used': 73722953728, u'op_per_sec': 2, u'write_bytes_sec': 0, u'version': 1268, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.652', u'bytes_avail': 2261789908992}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:19:59,367.367 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:00,086.086 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:03,870.870 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:04,589.589 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:05,522.522 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:06,265.265 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 840/52993 objects degraded (1.585%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7352539, u'degraded_objects': 840, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 73949778124, u'degraded_total': 52993, u'bytes_used': 73795211264, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1272, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.585', u'bytes_avail': 2261717651456}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:06,266.266 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:06,997.997 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:10,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:11,709.709 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:12,483.483 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:13,275.275 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 774/52825 objects degraded (1.465%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 774, u'num_pgs': 168, u'data_bytes': 73714897100, u'degraded_total': 52825, u'bytes_used': 74172612608, u'op_per_sec': 11, u'write_bytes_sec': 0, u'version': 1277, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.465', u'bytes_avail': 2261340250112}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:13,276.276 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:13,982.982 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:17,755.755 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:18,602.602 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:19,382.382 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:20,084.084 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 747/52702 objects degraded (1.417%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 747, u'num_pgs': 168, u'data_bytes': 73542930636, u'degraded_total': 52702, u'bytes_used': 74232057856, u'op_per_sec': 10, u'write_bytes_sec': 0, u'version': 1280, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.417', u'bytes_avail': 2261280804864}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:20,085.085 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:20,792.792 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:24,719.719 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:25,299.299 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:26,268.268 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:27,069.069 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:19:33.248525', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555696, u'health': u'HEALTH_OK', u'kb_used': 10773288, u'store_stats': {u'bytes_total': 4573217, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524625}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:19:33.497866', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437555404, u'health': u'HEALTH_OK', u'kb_used': 10773580, u'store_stats': {u'bytes_total': 5621829, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524661}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 656/52444 objects degraded (1.251%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4805285, u'degraded_objects': 656, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 73182220492, u'degraded_total': 52444, u'bytes_used': 74282659840, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1285, u'pgs_by_state': [{u'count': 160, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.251', u'bytes_avail': 2261230202880}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:27,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:27,839.839 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:31,645.645 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:32,379.379 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:33,169.169 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:33,961.961 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:19:33.965958', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436891644, u'health': u'HEALTH_OK', u'kb_used': 11437340, u'store_stats': {u'bytes_total': 5621923, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3524755}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 214/52234 objects degraded (0.410%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 214, u'num_pgs': 168, u'data_bytes': 72888619212, u'degraded_total': 52234, u'bytes_used': 74295222272, u'op_per_sec': 13, u'write_bytes_sec': 0, u'version': 1288, u'pgs_by_state': [{u'count': 161, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.410', u'bytes_avail': 2261217640448}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:33,962.962 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:34,669.669 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:38,441.441 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:39,063.063 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:40,009.009 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:40,780.780 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 16, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.009797', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.023833', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 157/51976 objects degraded (0.302%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8300469, u'degraded_objects': 157, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 72527909068, u'degraded_total': 51976, u'bytes_used': 74202300416, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 16, u'version': 1291, u'pgs_by_state': [{u'count': 161, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.302', u'bytes_avail': 2261310562304}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:40,782.782 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:41,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:45,310.310 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:46,031.031 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:46,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:47,584.584 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 133/51529 objects degraded (0.258%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5753320, u'degraded_objects': 133, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 71902957772, u'degraded_total': 51529, u'bytes_used': 73657806848, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 22, u'version': 1295, u'pgs_by_state': [{u'count': 161, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.258', u'bytes_avail': 2261855055872}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:47,585.585 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:48,403.403 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:52,325.325 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:52,963.963 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:53,906.906 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:20:54,679.679 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 100/51325 objects degraded (0.195%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 100, u'num_pgs': 168, u'data_bytes': 71617745100, u'degraded_total': 51325, u'bytes_used': 73684652032, u'op_per_sec': 22, u'write_bytes_sec': 0, u'version': 1299, u'pgs_by_state': [{u'count': 161, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.195', u'bytes_avail': 2261828210688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:20:54,680.680 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:20:55,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:20:59,184.184 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:00,431.431 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:01,381.381 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:02,113.113 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 107/51028 objects degraded (0.210%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9748980, u'degraded_objects': 107, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 71202509004, u'degraded_total': 51028, u'bytes_used': 73716797440, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1304, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.210', u'bytes_avail': 2261796065280}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:02,114.114 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:02,872.872 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:06,654.654 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:07,368.368 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:08,152.152 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:08,873.873 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 87/50722 objects degraded (0.172%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 87, u'num_pgs': 168, u'data_bytes': 70774689996, u'degraded_total': 50722, u'bytes_used': 73728126976, u'op_per_sec': 22, u'write_bytes_sec': 0, u'version': 1308, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.172', u'bytes_avail': 2261784735744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:08,874.874 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:09,585.585 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:11,860.860 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:21:11.784091 7f827bfff700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/7690 pipe(0x7f82a001d850 sd=13 :53101 s=2 pgs=7 cs=1 l=1 c=0x7f82a0021ed0).injecting socket failure 2014-04-01 16:21:13,517.517 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:14,294.294 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:15,084.084 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:15,791.791 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 39/50467 objects degraded (0.077%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14436362, u'degraded_objects': 39, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 70418174156, u'degraded_total': 50467, u'bytes_used': 73762963456, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1312, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.077', u'bytes_avail': 2261749899264}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:15,792.792 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:16,498.498 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:20,431.431 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:21,149.149 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:22,092.092 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:22,800.800 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12/50125 objects degraded (0.024%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6715658, u'degraded_objects': 12, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 69940023500, u'degraded_total': 50125, u'bytes_used': 73784324096, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 16, u'version': 1316, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'0.024', u'bytes_avail': 2261728538624}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:22,801.801 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:23,503.503 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:27,423.423 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:28,139.139 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:29,077.077 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:29,891.891 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:20:33.248813', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437519060, u'health': u'HEALTH_OK', u'kb_used': 10809924, u'store_stats': {u'bytes_total': 5741748, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:20:33.966235', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436853808, u'health': u'HEALTH_OK', u'kb_used': 11475176, u'store_stats': {u'bytes_total': 6790448, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:20:33.498208', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437518972, u'health': u'HEALTH_OK', u'kb_used': 10810012, u'store_stats': {u'bytes_total': 6790373, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -33/49897 objects degraded (-0.066%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10285858, u'degraded_objects': 18446744073709551583L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 69621256396, u'degraded_total': 49897, u'bytes_used': 73807568896, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1320, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.066', u'bytes_avail': 2261705293824}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:29,892.892 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:30,824.824 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:35,045.045 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:35,602.602 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:36,542.542 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:37,252.252 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -53/49618 objects degraded (-0.107%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9554530, u'degraded_objects': 18446744073709551563L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 69231186124, u'degraded_total': 49618, u'bytes_used': 73821040640, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1324, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.107', u'bytes_avail': 2261691822080}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:37,252.252 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:37,824.824 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:21:37.748633 7f82981e0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6804/5446 pipe(0x7f82a0011140 sd=12 :37215 s=2 pgs=7 cs=1 l=1 c=0x7f82a000c150).injecting socket failure 2014-04-01 16:21:37,950.950 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:41,723.723 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:42,446.446 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:43,428.428 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:44,118.118 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -74/49312 objects degraded (-0.150%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551542L, u'num_pgs': 168, u'data_bytes': 68803367116, u'degraded_total': 49312, u'bytes_used': 73846669312, u'op_per_sec': 19, u'write_bytes_sec': 0, u'version': 1328, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.150', u'bytes_avail': 2261666193408}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:44,119.119 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:44,915.915 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:48,839.839 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:49,572.572 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:50,365.365 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:51,186.186 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -111/49114 objects degraded (-0.226%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8182691, u'degraded_objects': 18446744073709551505L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 68526543052, u'degraded_total': 49114, u'bytes_used': 73880342528, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1332, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.226', u'bytes_avail': 2261632520192}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:51,187.187 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:51,889.889 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:56,613.613 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:57,174.174 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:21:57,962.962 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:21:58,726.726 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -136/48769 objects degraded (-0.279%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551480L, u'num_pgs': 168, u'data_bytes': 68044198092, u'degraded_total': 48769, u'bytes_used': 73897365504, u'op_per_sec': 14, u'write_bytes_sec': 0, u'version': 1336, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.279', u'bytes_avail': 2261615497216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:21:58,727.727 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:21:59,504.504 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:03,423.423 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:04,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:04,994.994 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:05,812.812 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -183/48544 objects degraded (-0.377%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5571739, u'degraded_objects': 18446744073709551433L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 67729625292, u'degraded_total': 48544, u'bytes_used': 73923874816, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1339, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.377', u'bytes_avail': 2261588987904}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:05,813.813 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:06,517.517 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:10,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:11,020.020 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:12,626.626 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:13,372.372 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -206/48220 objects degraded (-0.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551410L, u'num_pgs': 168, u'data_bytes': 67276640460, u'degraded_total': 48220, u'bytes_used': 74030379008, u'op_per_sec': 13, u'write_bytes_sec': 0, u'version': 1344, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.427', u'bytes_avail': 2261482483712}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:13,373.373 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:14,072.072 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:17,831.831 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:18,608.608 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:19,539.539 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:20,265.265 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -247/47965 objects degraded (-0.515%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11697325, u'degraded_objects': 18446744073709551369L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 66920124620, u'degraded_total': 47965, u'bytes_used': 74046169088, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1348, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.515', u'bytes_avail': 2261466693632}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:20,266.266 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:21,033.033 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:24,807.807 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:25,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:26,463.463 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:27,348.348 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:21:33.249068', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473492, u'health': u'HEALTH_OK', u'kb_used': 10855492, u'store_stats': {u'bytes_total': 7838900, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644580}}, {u'last_updated': u'2014-04-01 07:21:33.966499', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436819308, u'health': u'HEALTH_OK', u'kb_used': 11509676, u'store_stats': {u'bytes_total': 8887600, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644704}}, {u'last_updated': u'2014-04-01 07:21:33.498429', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437473304, u'health': u'HEALTH_OK', u'kb_used': 10855680, u'store_stats': {u'bytes_total': 8887525, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3644629}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -270/47554 objects degraded (-0.568%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4759136, u'degraded_objects': 18446744073709551346L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 66345504972, u'degraded_total': 47554, u'bytes_used': 74072596480, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 16, u'version': 1352, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.568', u'bytes_avail': 2261440266240}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:27,349.349 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:28,057.057 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:31,832.832 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:32,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:33,337.337 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:34,107.107 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -292/47377 objects degraded (-0.616%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551324L, u'num_pgs': 168, u'data_bytes': 66098041036, u'degraded_total': 47377, u'bytes_used': 74087923712, u'op_per_sec': 14, u'write_bytes_sec': 0, u'version': 1355, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.616', u'bytes_avail': 2261424939008}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:34,108.108 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:34,825.825 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:38,752.752 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:39,369.369 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:40,334.334 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:41,134.134 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -332/47161 objects degraded (-0.704%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8751716, u'degraded_objects': 18446744073709551284L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 65796051148, u'degraded_total': 47161, u'bytes_used': 74121826304, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 12, u'version': 1359, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.704', u'bytes_avail': 2261391036416}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:41,135.135 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:41,846.846 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:45,623.623 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:46,351.351 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:47,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:47,964.964 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -357/46789 objects degraded (-0.763%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551259L, u'num_pgs': 168, u'data_bytes': 65275957452, u'degraded_total': 46789, u'bytes_used': 74137853952, u'op_per_sec': 16, u'write_bytes_sec': 0, u'version': 1363, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.763', u'bytes_avail': 2261375008768}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:47,965.965 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:48,726.726 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:52,500.500 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:53,219.219 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:22:54,328.328 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:22:58,259.259 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -408/46393 objects degraded (-0.879%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551208L, u'num_pgs': 168, u'data_bytes': 64722309324, u'degraded_total': 46393, u'bytes_used': 74152574976, u'op_per_sec': 16, u'write_bytes_sec': 0, u'version': 1369, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.879', u'bytes_avail': 2261360287744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:22:58,260.260 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:22:59,119.119 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:03,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:03,865.865 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:04,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:05,603.603 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -439/46162 objects degraded (-0.951%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7422870, u'degraded_objects': 18446744073709551177L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 64399347916, u'degraded_total': 46162, u'bytes_used': 74193301504, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1373, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.951', u'bytes_avail': 2261319561216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:05,604.604 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:06,302.302 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:10,086.086 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:10,808.808 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:11,593.593 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:12,306.306 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -465/45859 objects degraded (-1.014%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10004156, u'degraded_objects': 18446744073709551151L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 63975723212, u'degraded_total': 45859, u'bytes_used': 74213535744, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 13, u'version': 1377, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.014', u'bytes_avail': 2261299326976}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:12,307.307 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:13,004.004 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:16,927.927 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:17,704.704 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:18,652.652 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:19,460.460 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -490/45583 objects degraded (-1.075%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551126L, u'num_pgs': 168, u'data_bytes': 63589847244, u'degraded_total': 45583, u'bytes_used': 74227486720, u'op_per_sec': 16, u'write_bytes_sec': 0, u'version': 1381, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.075', u'bytes_avail': 2261285376000}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:19,461.461 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:20,288.288 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:24,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:25,302.302 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:26,545.545 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:27,404.404 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:22:33.249270', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437439228, u'health': u'HEALTH_OK', u'kb_used': 10889756, u'store_stats': {u'bytes_total': 5520404, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423236}}, {u'last_updated': u'2014-04-01 07:22:33.966784', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436791372, u'health': u'HEALTH_OK', u'kb_used': 11537612, u'store_stats': {u'bytes_total': 6569079, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423335}}, {u'last_updated': u'2014-04-01 07:22:33.498632', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437438976, u'health': u'HEALTH_OK', u'kb_used': 10890008, u'store_stats': {u'bytes_total': 6569030, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3423286}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -563/45283 objects degraded (-1.243%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8800071, u'degraded_objects': 18446744073709551053L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 63170416844, u'degraded_total': 45283, u'bytes_used': 74258890752, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1386, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.243', u'bytes_avail': 2261253971968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:27,405.405 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:28,187.187 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:32,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:32,753.753 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:33,696.696 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:34,410.410 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -595/45055 objects degraded (-1.321%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551021L, u'num_pgs': 168, u'data_bytes': 62851649740, u'degraded_total': 45055, u'bytes_used': 74277752832, u'op_per_sec': 20, u'write_bytes_sec': 0, u'version': 1390, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.321', u'bytes_avail': 2261235109888}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:34,411.411 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:35,166.166 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:38,948.948 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:39,728.728 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:40,514.514 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:41,299.299 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -638/44821 objects degraded (-1.423%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7542165, u'degraded_objects': 18446744073709550978L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 62524494028, u'degraded_total': 44821, u'bytes_used': 74312187904, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 15, u'version': 1394, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.423', u'bytes_avail': 2261200674816}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:41,301.301 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:42,019.019 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:45,969.969 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:46,749.749 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:47,535.535 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:48,322.322 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -666/44512 objects degraded (-1.496%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550950L, u'num_pgs': 168, u'data_bytes': 62092480716, u'degraded_total': 44512, u'bytes_used': 74338381824, u'op_per_sec': 15, u'write_bytes_sec': 0, u'version': 1399, u'pgs_by_state': [{u'count': 162, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.496', u'bytes_avail': 2261174480896}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:48,323.323 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:49,097.097 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:53,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:53,620.620 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:23:54,705.705 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:23:55,488.488 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -507/44308 objects degraded (-1.144%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9864352, u'degraded_objects': 18446744073709551109L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 61807268044, u'degraded_total': 44308, u'bytes_used': 74361499648, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1403, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.144', u'bytes_avail': 2261151363072}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:23:55,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:23:56,253.253 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:00,015.015 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:00,811.811 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:01,599.599 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:02,367.367 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -526/44092 objects degraded (-1.193%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8014561, u'degraded_objects': 18446744073709551090L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 61505278156, u'degraded_total': 44092, u'bytes_used': 74380394496, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1406, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.193', u'bytes_avail': 2261132468224}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:02,368.368 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:03,129.129 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:06,896.896 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:07,672.672 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:08,605.605 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:09,325.325 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -548/43822 objects degraded (-1.251%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551068L, u'num_pgs': 168, u'data_bytes': 61127790796, u'degraded_total': 43822, u'bytes_used': 74387726336, u'op_per_sec': 14, u'write_bytes_sec': 0, u'version': 1410, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.251', u'bytes_avail': 2261125136384}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:09,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:10,064.064 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:14,008.008 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:14,733.733 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:15,672.672 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:16,443.443 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -583/43573 objects degraded (-1.338%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5013697, u'degraded_objects': 18446744073709551033L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 60779663564, u'degraded_total': 43573, u'bytes_used': 74418450432, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 12, u'version': 1414, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.338', u'bytes_avail': 2261094412288}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:16,444.444 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:18,028.028 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:21,803.803 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:22,577.577 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:23,521.521 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:24,230.230 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -610/43258 objects degraded (-1.410%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551006L, u'num_pgs': 168, u'data_bytes': 60339261644, u'degraded_total': 43258, u'bytes_used': 74434588672, u'op_per_sec': 12, u'write_bytes_sec': 0, u'version': 1418, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.410', u'bytes_avail': 2261078274048}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:24,232.232 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:25,082.082 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:29,004.004 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:29,740.740 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:30,663.663 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:31,448.448 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:23:33.249564', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399568, u'health': u'HEALTH_OK', u'kb_used': 10929416, u'store_stats': {u'bytes_total': 5130745, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:23:33.967049', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436762996, u'health': u'HEALTH_OK', u'kb_used': 11565988, u'store_stats': {u'bytes_total': 6179671, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:23:33.511655', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437399284, u'health': u'HEALTH_OK', u'kb_used': 10929700, u'store_stats': {u'bytes_total': 6179556, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -652/43045 objects degraded (-1.515%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5566108, u'degraded_objects': 18446744073709550964L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 60041466060, u'degraded_total': 43045, u'bytes_used': 74464559104, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 13, u'version': 1421, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.515', u'bytes_avail': 2261048303616}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:31,449.449 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:32,151.151 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:35,916.916 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:36,636.636 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:37,563.563 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:38,285.285 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -667/42673 objects degraded (-1.563%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550949L, u'num_pgs': 168, u'data_bytes': 59521372364, u'degraded_total': 42673, u'bytes_used': 74483003392, u'op_per_sec': 18, u'write_bytes_sec': 0, u'version': 1426, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.563', u'bytes_avail': 2261029859328}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:38,286.286 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:39,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:42,972.972 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:43,601.601 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:44,631.631 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:45,409.409 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -708/42445 objects degraded (-1.668%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5594221, u'degraded_objects': 18446744073709550908L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 59202605260, u'degraded_total': 42445, u'bytes_used': 74520354816, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1429, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.668', u'bytes_avail': 2260992507904}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:45,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:46,204.204 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:50,146.146 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:51,116.116 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:52,527.527 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:24:53,566.566 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -730/42145 objects degraded (-1.732%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550886L, u'num_pgs': 168, u'data_bytes': 58783174860, u'degraded_total': 42145, u'bytes_used': 74537893888, u'op_per_sec': 18, u'write_bytes_sec': 0, u'version': 1434, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.732', u'bytes_avail': 2260974968832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:24:53,567.567 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:54,929.929 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:24:59,115.115 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:24:59,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:00,986.986 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:01,838.838 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -762/41839 objects degraded (-1.821%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7540186, u'degraded_objects': 18446744073709550854L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 58355355852, u'degraded_total': 41839, u'bytes_used': 74562879488, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1439, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.821', u'bytes_avail': 2260949983232}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:01,840.840 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:02,791.791 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:06,840.840 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:07,562.562 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:08,497.497 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:09,203.203 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -781/41545 objects degraded (-1.880%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550835L, u'num_pgs': 168, u'data_bytes': 57944314060, u'degraded_total': 41545, u'bytes_used': 74586304512, u'op_per_sec': 14, u'write_bytes_sec': 0, u'version': 1443, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.880', u'bytes_avail': 2260926558208}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:09,204.204 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:09,905.905 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:13,680.680 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:14,455.455 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:15,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:16,276.276 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -831/41317 objects degraded (-2.011%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8783186, u'degraded_objects': 18446744073709550785L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 57625546956, u'degraded_total': 41317, u'bytes_used': 74611249152, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 12, u'version': 1447, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.011', u'bytes_avail': 2260901613568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:16,277.277 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:17,038.038 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:20,957.957 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:21,747.747 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:22,691.691 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:24,500.500 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -846/40993 objects degraded (-2.064%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550770L, u'num_pgs': 168, u'data_bytes': 57172562124, u'degraded_total': 40993, u'bytes_used': 74619236352, u'op_per_sec': 15, u'write_bytes_sec': 0, u'version': 1451, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.064', u'bytes_avail': 2260893626368}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:24,501.501 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:25,356.356 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:26,247.247 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:25:26.170102 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a000edc0 sd=11 :58907 s=2 pgs=5 cs=1 l=1 c=0x7f82a000f020).injecting socket failure 2014-04-01 16:25:29,283.283 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:30,229.229 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:31,322.322 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:31,966.966 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:25:31.826922 7f82984e3700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/7690 pipe(0x7f82a0023510 sd=14 :53146 s=2 pgs=9 cs=1 l=1 c=0x7f82a00024f0).injecting socket failure 2014-04-01 16:25:31,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:25:31.890786 7f6c3a74a700 0 monclient: hunting for new mon 2014-04-01 16:25:32,110.110 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:24:33.249878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356384, u'health': u'HEALTH_OK', u'kb_used': 10972600, u'store_stats': {u'bytes_total': 6179321, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:24:33.967317', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436732288, u'health': u'HEALTH_OK', u'kb_used': 11596696, u'store_stats': {u'bytes_total': 8276823, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:24:33.511932', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437356168, u'health': u'HEALTH_OK', u'kb_used': 10972816, u'store_stats': {u'bytes_total': 8276708, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -893/40705 objects degraded (-2.194%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7462477, u'degraded_objects': 18446744073709550723L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 56769908940, u'degraded_total': 40705, u'bytes_used': 74671116288, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1455, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.194', u'bytes_avail': 2260841746432}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:32,111.111 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:32,995.995 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:37,420.420 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:37,452.452 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:25:37.375815 7f82987e6700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a000be20 sd=10 :33793 s=2 pgs=13 cs=1 l=1 c=0x7f82a000d740).injecting socket failure 2014-04-01 16:25:38,200.200 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:39,146.146 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:39,869.869 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 18, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.008001', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.019209', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -908/40411 objects degraded (-2.247%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550708L, u'num_pgs': 168, u'data_bytes': 56358867148, u'degraded_total': 40411, u'bytes_used': 74675990528, u'op_per_sec': 18, u'write_bytes_sec': 0, u'version': 1459, u'pgs_by_state': [{u'count': 163, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.247', u'bytes_avail': 2260836872192}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:39,870.870 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:40,569.569 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:44,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:45,053.053 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:45,985.985 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:46,777.777 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -759/40111 objects degraded (-1.892%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5549217, u'degraded_objects': 18446744073709550857L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 55939436748, u'degraded_total': 40111, u'bytes_used': 74707951616, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1463, u'pgs_by_state': [{u'count': 164, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.892', u'bytes_avail': 2260804911104}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:46,778.778 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:47,475.475 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:51,398.398 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:52,129.129 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:53,820.820 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:25:54,732.732 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -797/39817 objects degraded (-2.002%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7526831, u'degraded_objects': 18446744073709550819L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 55528394956, u'degraded_total': 39817, u'bytes_used': 74750316544, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1467, u'pgs_by_state': [{u'count': 164, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.002', u'bytes_avail': 2260762546176}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:25:54,733.733 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:25:55,658.658 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:25:59,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:00,377.377 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:01,357.357 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:02,087.087 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -818/39655 objects degraded (-2.063%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11278254, u'degraded_objects': 18446744073709550798L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 55301902540, u'degraded_total': 39655, u'bytes_used': 74764824576, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1470, u'pgs_by_state': [{u'count': 164, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.063', u'bytes_avail': 2260748038144}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:02,088.088 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:02,786.786 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:06,552.552 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:07,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:08,103.103 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:08,890.890 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -838/39301 objects degraded (-2.132%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709550778L, u'num_pgs': 168, u'data_bytes': 54806974668, u'degraded_total': 39301, u'bytes_used': 74777223168, u'op_per_sec': 17, u'write_bytes_sec': 0, u'version': 1474, u'pgs_by_state': [{u'count': 164, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.132', u'bytes_avail': 2260735639552}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:08,891.891 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:09,595.595 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:13,377.377 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:14,148.148 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:14,930.930 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:15,731.731 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -866/39073 objects degraded (-2.216%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7354864, u'degraded_objects': 18446744073709550750L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 54488207564, u'degraded_total': 39073, u'bytes_used': 74805895168, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1478, u'pgs_by_state': [{u'count': 164, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-2.216', u'bytes_avail': 2260706967552}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:15,732.732 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:16,429.429 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:20,354.354 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:21,076.076 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:22,022.022 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:22,746.746 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -674/38815 objects degraded (-1.736%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9854263, u'degraded_objects': 18446744073709550942L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 54127497420, u'degraded_total': 38815, u'bytes_used': 74807353344, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1482, u'pgs_by_state': [{u'count': 165, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.736', u'bytes_avail': 2260705509376}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:22,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:23,457.457 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:27,370.370 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:28,153.153 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:29,476.476 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:30,165.165 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:25:33.250137', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437313016, u'health': u'HEALTH_OK', u'kb_used': 11015968, u'store_stats': {u'bytes_total': 8276473, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:25:33.967587', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700208, u'health': u'HEALTH_OK', u'kb_used': 11628776, u'store_stats': {u'bytes_total': 10373975, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:25:33.512178', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437312936, u'health': u'HEALTH_OK', u'kb_used': 11016048, u'store_stats': {u'bytes_total': 10373860, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -518/38500 objects degraded (-1.345%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9355475, u'degraded_objects': 18446744073709551098L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 53687095500, u'degraded_total': 38500, u'bytes_used': 74851577856, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 13, u'version': 1486, u'pgs_by_state': [{u'count': 166, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.345', u'bytes_avail': 2260661284864}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:30,166.166 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:33,868.868 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:36,792.792 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:26:36.584204 7f82987e6700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a0007500 sd=14 :34958 s=2 pgs=14 cs=1 l=1 c=0x7f82a0007760).injecting socket failure 2014-04-01 16:26:37,799.799 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:38,521.521 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:39,296.296 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:40,076.076 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -367/38113 objects degraded (-0.963%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12327058, u'degraded_objects': 18446744073709551249L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 53146030284, u'degraded_total': 38113, u'bytes_used': 74903224320, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 14, u'version': 1492, u'pgs_by_state': [{u'count': 167, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.963', u'bytes_avail': 2260609638400}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:40,076.076 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:40,795.795 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:44,560.560 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:45,281.281 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:46,060.060 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:46,782.782 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -374/37891 objects degraded (-0.987%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6299075, u'degraded_objects': 18446744073709551242L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 52835651788, u'degraded_total': 37891, u'bytes_used': 74906161152, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1496, u'pgs_by_state': [{u'count': 167, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.987', u'bytes_avail': 2260606701568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:46,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:47,587.587 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:51,349.349 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:52,298.298 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:53,496.496 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:26:54,336.336 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -385/37645 objects degraded (-1.023%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551231L, u'num_pgs': 168, u'data_bytes': 52491718860, u'degraded_total': 37645, u'bytes_used': 74932707328, u'op_per_sec': 19, u'write_bytes_sec': 0, u'version': 1500, u'pgs_by_state': [{u'count': 167, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.023', u'bytes_avail': 2260580155392}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:26:54,337.337 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:55,168.168 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:26:59,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:26:59,804.804 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:00,900.900 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:27:01,670.670 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -426/37429 objects degraded (-1.138%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13694946, u'degraded_objects': 18446744073709551190L, u'num_pgs': 168, u'recovering_keys_per_sec': 0, u'data_bytes': 52189728972, u'degraded_total': 37429, u'bytes_used': 74968956928, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1504, u'pgs_by_state': [{u'count': 167, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.138', u'bytes_avail': 2260543905792}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:27:01,671.671 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:02,482.482 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:06,400.400 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:07,175.175 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:07,960.960 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:27:08,744.744 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -439/37078 objects degraded (-1.184%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18446744073709551177L, u'num_pgs': 168, u'data_bytes': 51698995404, u'degraded_total': 37078, u'bytes_used': 74998296576, u'op_per_sec': 15, u'write_bytes_sec': 0, u'version': 1508, u'pgs_by_state': [{u'count': 167, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-1.184', u'bytes_avail': 2260514566144}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:27:08,745.745 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:09,444.444 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:13,211.211 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:13,985.985 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:14,764.764 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:27:15,485.485 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 20, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.006559', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.015499', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:26:33.250388', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437272004, u'health': u'HEALTH_OK', u'kb_used': 11056980, u'store_stats': {u'bytes_total': 10373625, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033577}}, {u'last_updated': u'2014-04-01 07:26:33.967867', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436666668, u'health': u'HEALTH_OK', u'kb_used': 11662316, u'store_stats': {u'bytes_total': 13519703, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033927}}, {u'last_updated': u'2014-04-01 07:26:33.512457', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437271884, u'health': u'HEALTH_OK', u'kb_used': 11057100, u'store_stats': {u'bytes_total': 13519588, u'bytes_log': 10420224, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3033812}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 46 > pgp_num 26'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'num_pgs': 168, u'data_bytes': 51342479564, u'bytes_used': 75009994752, u'op_per_sec': 14, u'version': 1511, u'pgs_by_state': [{u'count': 168, u'state_name': u'active+clean'}], u'write_bytes_sec': 0, u'bytes_avail': 2260502867968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 5}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:27:15,486.486 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 16:27:15,486.486 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:15,487.487 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.771241 2014-04-01 16:27:15,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.771241360368' 2014-04-01 16:27:16,783.783 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 0.771241 (8505442) 2014-04-01 16:27:21,795.795 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 3, 2, 4] 2014-04-01 16:27:21,796.796 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:21,796.796 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:27:21,797.797 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:22,588.588 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:23,522.522 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 16:27:23,522.522 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 56' 2014-04-01 16:27:24,766.766 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:27:24.687597 7f82987e6700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a000f150 sd=10 :34989 s=2 pgs=15 cs=1 l=1 c=0x7f82a0013a70).injecting socket failure 2014-04-01 16:27:25,001.001 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 56 2014-04-01 16:27:30,010.010 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 3, 2, 4] 2014-04-01 16:27:30,011.011 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:30,011.011 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:27:30,012.012 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:27:31,166.166 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:27:37,054.054 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:27:36.975499 7f82983e2700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/7690 pipe(0x7f82a0007840 sd=13 :53320 s=2 pgs=11 cs=1 l=1 c=0x7f82a000c7e0).injecting socket failure 2014-04-01 16:27:37,593.593 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 5, 3, 2, 4] 2014-04-01 16:27:37,594.594 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:37,595.595 INFO:teuthology.task.thrashosds.thrasher:Killing osd 5, live_osds are [1, 0, 5, 3, 2, 4] 2014-04-01 16:27:37,596.596 DEBUG:teuthology.task.ceph.osd.5:waiting for process to exit 2014-04-01 16:27:37,773.773 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:27:37.693988 7f82992e8700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6810/4980 pipe(0x7f82a0007840 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a0020030).fault 2014-04-01 16:27:37,775.775 INFO:teuthology.task.ceph.osd.5:Stopped 2014-04-01 16:27:42,775.775 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [5] live_osds: [1, 0, 3, 2, 4] 2014-04-01 16:27:42,776.776 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:42,777.777 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 5 2014-04-01 16:27:42,777.777 INFO:teuthology.task.ceph.osd.5:Restarting 2014-04-01 16:27:42,778.778 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 5' 2014-04-01 16:27:43,101.101 INFO:teuthology.task.ceph.osd.5:Started 2014-04-01 16:27:43,102.102 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight' 2014-04-01 16:27:43,373.373 INFO:teuthology.task.ceph.osd.5.out:[10.214.133.38]: starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal 2014-04-01 16:27:43,588.588 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:27:43,588.588 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:27:43,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:27:43,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:27:43,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:27:43,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:27:43,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:27:43,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:27:43,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:27:43,598.598 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:27:43,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:27:43,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:27:43,600.600 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 5, ['dump_ops_in_flight'] 2014-04-01 16:27:43,903.903 INFO:teuthology.task.ceph.osd.5.err:[10.214.133.38]: 2014-04-01 07:27:43.826990 7fb9a9f57780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 16:27:48,594.594 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight' 2014-04-01 16:27:54,023.023 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 2, 4, 5] 2014-04-01 16:27:54,024.024 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:54,027.027 INFO:teuthology.task.thrashosds.thrasher:Killing osd 1, live_osds are [1, 0, 3, 2, 4, 5] 2014-04-01 16:27:54,027.027 DEBUG:teuthology.task.ceph.osd.1:waiting for process to exit 2014-04-01 16:27:54,214.214 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:27:54.137929 7f82992e8700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6805/4943 pipe(0x7f82a000d000 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a000d260).fault 2014-04-01 16:27:54,220.220 INFO:teuthology.task.ceph.osd.1:Stopped 2014-04-01 16:27:59,218.218 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:27:59,218.218 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:27:59,221.221 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 16:27:59,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:28:00,000.000 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:28:00,963.963 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pgp_num 56' 2014-04-01 16:28:01,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pgp_num to 56 2014-04-01 16:28:06,993.993 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:06,994.994 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:06,995.995 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 4 2014-04-01 16:28:06,995.995 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:28:06,995.995 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:28:06,996.996 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-04-01 16:28:12,429.429 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3, 2] out_osds: [5] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:12,430.430 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:12,433.433 INFO:teuthology.task.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 4, 3, 2] 2014-04-01 16:28:12,433.433 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-04-01 16:28:14,197.197 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.2. 2014-04-01 16:28:19,216.216 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:19,217.217 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:19,217.217 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 16:28:19,218.218 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:28:19,962.962 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:28:21,062.062 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 34' 2014-04-01 16:28:22,567.567 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pgp_num to 34 2014-04-01 16:28:27,567.567 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:27,568.568 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:27,568.568 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 16:28:27,569.569 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:28:28,470.470 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:28:29,403.403 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 34' 2014-04-01 16:28:30,664.664 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pgp_num to 34 2014-04-01 16:28:35,663.663 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:35,663.663 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:35,663.663 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 16:28:35,663.663 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:28:35,663.663 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:28:35,663.663 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 16:28:41,101.101 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:41,101.101 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:41,102.102 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:28:41,103.103 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:28:42,038.038 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:28:42,974.974 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 16:28:42,974.974 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 66' 2014-04-01 16:28:43,905.905 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 66 2014-04-01 16:28:48,917.917 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1] live_osds: [0, 3, 2, 4, 5] 2014-04-01 16:28:48,918.918 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:48,918.918 INFO:teuthology.task.thrashosds.thrasher:Killing osd 4, live_osds are [0, 3, 2, 4, 5] 2014-04-01 16:28:48,919.919 DEBUG:teuthology.task.ceph.osd.4:waiting for process to exit 2014-04-01 16:28:49,121.121 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:28:49.043861 7f82992e8700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6800/7690 pipe(0x7f82a0021e30 sd=8 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a00096c0).fault 2014-04-01 16:28:49,125.125 INFO:teuthology.task.ceph.osd.4:Stopped 2014-04-01 16:28:54,124.124 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 3, 2, 5] 2014-04-01 16:28:54,125.125 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:28:54,127.127 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 16:28:54,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:28:55,071.071 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:29:00,987.987 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 3, 2, 5] 2014-04-01 16:29:00,988.988 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:00,989.989 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 16:29:00,989.989 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:29:01,770.770 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:29:07,690.690 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 3, 2, 5] 2014-04-01 16:29:07,691.691 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:07,692.692 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 16:29:07,692.692 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:29:08,568.568 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:29:14,644.644 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 3, 2, 5] 2014-04-01 16:29:14,645.645 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:14,645.645 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 16:29:14,646.646 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:29:14,646.646 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:29:14,646.646 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 16:29:20,073.073 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 3, 2, 5] 2014-04-01 16:29:20,074.074 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:20,076.076 INFO:teuthology.task.thrashosds.thrasher:Killing osd 3, live_osds are [0, 3, 2, 5] 2014-04-01 16:29:20,077.077 DEBUG:teuthology.task.ceph.osd.3:waiting for process to exit 2014-04-01 16:29:20,248.248 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:29:20.172188 7f82986e5700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6804/5446 pipe(0x7f82a000f150 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f82a0010290).fault 2014-04-01 16:29:20,254.254 INFO:teuthology.task.ceph.osd.3:Stopped 2014-04-01 16:29:25,252.252 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4, 3] live_osds: [0, 2, 5] 2014-04-01 16:29:25,253.253 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:25,256.256 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool data 2014-04-01 16:29:25,256.256 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:29:26,030.030 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:29:27,121.121 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pgp_num 34' 2014-04-01 16:29:28,214.214 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pgp_num to 34 2014-04-01 16:29:33,213.213 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4, 3] live_osds: [0, 2, 5] 2014-04-01 16:29:33,213.213 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:33,214.214 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 3 2014-04-01 16:29:33,214.214 INFO:teuthology.task.ceph.osd.3:Restarting 2014-04-01 16:29:33,215.215 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-04-01 16:29:33,536.536 INFO:teuthology.task.ceph.osd.3:Started 2014-04-01 16:29:33,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-04-01 16:29:33,588.588 INFO:teuthology.task.ceph.osd.3.out:[10.214.133.38]: starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-04-01 16:29:33,973.973 INFO:teuthology.task.ceph.osd.3.err:[10.214.133.38]: 2014-04-01 07:29:33.897098 7f9e7bd4e780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 16:29:33,989.989 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:29:33,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:29:33,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:29:33,990.990 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:29:33,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:29:33,991.991 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:29:33,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:29:33,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:29:33,992.992 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:29:33,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:29:33,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:29:33,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:29:33,994.994 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 3, ['dump_ops_in_flight'] 2014-04-01 16:29:38,995.995 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight' 2014-04-01 16:29:44,437.437 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 4, 3] out_osds: [5, 2] dead_osds: [1, 4] live_osds: [0, 2, 5, 3] 2014-04-01 16:29:44,438.438 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:44,439.439 INFO:teuthology.task.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 4, 3] 2014-04-01 16:29:44,439.439 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 4' 2014-04-01 16:29:45,512.512 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.4. 2014-04-01 16:29:50,510.510 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3] out_osds: [5, 2, 4] dead_osds: [1, 4] live_osds: [0, 2, 5, 3] 2014-04-01 16:29:50,510.510 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:50,511.511 INFO:teuthology.task.thrashosds.thrasher:Adding osd 2 2014-04-01 16:29:50,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-04-01 16:29:52,030.030 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.2. 2014-04-01 16:29:52,051.051 INFO:teuthology.task.thrashosds.thrasher:Added osd 2 2014-04-01 16:29:57,049.049 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [1, 4] live_osds: [0, 2, 5, 3] 2014-04-01 16:29:57,050.050 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:29:57,051.051 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 1 2014-04-01 16:29:57,051.051 INFO:teuthology.task.ceph.osd.1:Restarting 2014-04-01 16:29:57,051.051 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' 2014-04-01 16:29:57,373.373 INFO:teuthology.task.ceph.osd.1:Started 2014-04-01 16:29:57,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight' 2014-04-01 16:29:57,529.529 INFO:teuthology.task.ceph.osd.1.out:[10.214.133.37]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2014-04-01 16:29:57,792.792 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 16:29:57,793.793 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 16:29:57,793.793 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 16:29:57,793.793 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 16:29:57,794.794 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 16:29:57,794.794 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 16:29:57,794.794 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 16:29:57,794.794 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 16:29:57,795.795 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 16:29:57,795.795 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 16:29:57,795.795 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 16:29:57,799.799 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 16:29:57,800.800 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 1, ['dump_ops_in_flight'] 2014-04-01 16:29:58,035.035 INFO:teuthology.task.ceph.osd.1.err:[10.214.133.37]: 2014-04-01 07:29:57.956165 7f9e7efe0780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 16:30:02,798.798 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight' 2014-04-01 16:30:03,214.214 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 16:30:03,215.215 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 16:30:03,215.215 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 16:30:03,215.215 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 16:30:03,216.216 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 16:30:03,216.216 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 16:30:03,216.216 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 16:30:03,222.222 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 16:30:03,222.222 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 16:30:03,222.222 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 16:30:03,223.223 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 16:30:03,223.223 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 16:30:03,224.224 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 1, ['dump_ops_in_flight'] 2014-04-01 16:30:08,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight' 2014-04-01 16:30:13,654.654 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [4] live_osds: [0, 2, 5, 3, 1] 2014-04-01 16:30:13,655.655 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:30:13,656.656 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 16:30:13,656.656 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:30:13,656.656 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:30:13,657.657 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 16:30:19,096.096 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [4] live_osds: [0, 2, 5, 3, 1] 2014-04-01 16:30:19,097.097 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:30:19,100.100 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 16:30:19,101.101 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 16:30:19,101.101 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 16:30:19,102.102 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 16:30:24,559.559 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [4] live_osds: [0, 2, 5, 3, 1] 2014-04-01 16:30:24,560.560 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:30:24,560.560 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 16:30:24,561.561 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:30:25,198.198 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:30:26,440.440 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 16:30:26,440.440 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 76' 2014-04-01 16:30:28,070.070 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 76 2014-04-01 16:30:33,081.081 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [4] live_osds: [0, 2, 5, 3, 1] 2014-04-01 16:30:33,082.082 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:30:33,083.083 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 16:30:33,083.083 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:30:33,640.640 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:30:39,878.878 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [4] live_osds: [0, 2, 5, 3, 1] 2014-04-01 16:30:39,878.878 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 4 2014-04-01 16:30:39,879.879 INFO:teuthology.task.ceph.osd.4:Restarting 2014-04-01 16:30:39,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4' 2014-04-01 16:30:40,301.301 INFO:teuthology.task.ceph.osd.4:Started 2014-04-01 16:30:40,301.301 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:30:40,353.353 INFO:teuthology.task.ceph.osd.4.out:[10.214.133.38]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal 2014-04-01 16:30:40,671.671 INFO:teuthology.task.ceph.osd.4.err:[10.214.133.38]: 2014-04-01 07:30:40.594725 7f2d8a953780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 16:30:40,722.722 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:30:40,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:30:40,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:30:40,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:30:40,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:30:40,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:30:40,724.724 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:30:40,724.724 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:30:40,724.724 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:30:40,730.730 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:30:40,731.731 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:30:40,731.731 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:30:40,731.731 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:30:45,730.730 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:30:46,145.145 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:30:46,146.146 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:30:46,146.146 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:30:46,153.153 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:30:46,153.153 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:30:46,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:30:46,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:30:46,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:30:46,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:30:46,154.154 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:30:46,155.155 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:30:46,155.155 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:30:46,155.155 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:30:51,151.151 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:30:51,575.575 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 16:30:51,575.575 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 16:30:51,575.575 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 16:30:51,576.576 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 16:30:51,576.576 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 16:30:51,576.576 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 16:30:51,576.576 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 16:30:51,577.577 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 16:30:51,577.577 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 16:30:51,577.577 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 16:30:51,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 16:30:51,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 16:30:51,581.581 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 16:30:56,581.581 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 16:30:57,017.017 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 16:30:57,018.018 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:30:57,776.776 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:30:59,030.030 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:00,038.038 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:01,143.143 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:01,925.925 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:30:33.251449', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074952, u'health': u'HEALTH_OK', u'kb_used': 11254032, u'store_stats': {u'bytes_total': 4988542, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:30:33.969024', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436511824, u'health': u'HEALTH_OK', u'kb_used': 11817160, u'store_stats': {u'bytes_total': 6037120, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:30:33.513510', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074936, u'health': u'HEALTH_OK', u'kb_used': 11254048, u'store_stats': {u'bytes_total': 6036983, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'32 pgs down'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9483/51642 objects degraded (18.363%); 57/17229 unfound (0.331%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 53561084, u'degraded_objects': 9483, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 72058147032, u'degraded_total': 51642, u'bytes_used': 59306340352, u'unfound_ratio': u'0.331', u'recovering_objects_per_sec': 12, u'write_bytes_sec': 1301946368, u'op_per_sec': 404, u'unfound_objects': 57, u'version': 1636, u'pgs_by_state': [{u'count': 1, u'state_name': u'inactive'}, {u'count': 10, u'state_name': u'active'}, {u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 123, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'down+remapped+peering'}, {u'count': 15, u'state_name': u'active+recovering'}, {u'count': 1, u'state_name': u'active+clean+replay'}, {u'count': 31, u'state_name': u'down+peering'}], u'degraded_ratio': u'18.363', u'bytes_avail': 1809103949824, u'unfound_total': 17229}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 131, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:01,926.926 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:02,688.688 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:06,771.771 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:07,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:08,619.619 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:09,468.468 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:30:33.251449', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074952, u'health': u'HEALTH_OK', u'kb_used': 11254032, u'store_stats': {u'bytes_total': 4988542, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:30:33.969024', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436511824, u'health': u'HEALTH_OK', u'kb_used': 11817160, u'store_stats': {u'bytes_total': 6037120, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:30:33.513510', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074936, u'health': u'HEALTH_OK', u'kb_used': 11254048, u'store_stats': {u'bytes_total': 6036983, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs down'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11843/54601 objects degraded (21.690%); 48/18216 unfound (0.264%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18250201, u'degraded_objects': 11843, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76189536472, u'degraded_total': 54601, u'bytes_used': 59333070848, u'unfound_ratio': u'0.264', u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 1, u'unfound_objects': 48, u'version': 1641, u'pgs_by_state': [{u'count': 14, u'state_name': u'active'}, {u'count': 25, u'state_name': u'active+recovery_wait'}, {u'count': 132, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}, {u'count': 1, u'state_name': u'remapped+peering'}, {u'count': 2, u'state_name': u'active+clean+replay'}, {u'count': 7, u'state_name': u'down+peering'}], u'degraded_ratio': u'21.690', u'bytes_avail': 1809077219328, u'unfound_total': 18216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 131, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:09,469.469 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:10,374.374 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:14,462.462 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:15,292.292 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:16,660.660 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:17,513.513 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:30:33.251449', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074952, u'health': u'HEALTH_OK', u'kb_used': 11254032, u'store_stats': {u'bytes_total': 4988542, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:30:33.969024', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436511824, u'health': u'HEALTH_OK', u'kb_used': 11817160, u'store_stats': {u'bytes_total': 6037120, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:30:33.513510', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074936, u'health': u'HEALTH_OK', u'kb_used': 11254048, u'store_stats': {u'bytes_total': 6036983, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 pgs down'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11494/54583 objects degraded (21.058%); 48/18210 unfound (0.264%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6578580, u'degraded_objects': 11494, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76164370648, u'degraded_total': 54583, u'bytes_used': 59350589440, u'unfound_ratio': u'0.264', u'recovering_objects_per_sec': 1, u'unfound_objects': 48, u'version': 1647, u'pgs_by_state': [{u'count': 12, u'state_name': u'active'}, {u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 133, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}, {u'count': 1, u'state_name': u'remapped+peering'}, {u'count': 3, u'state_name': u'active+clean+replay'}, {u'count': 6, u'state_name': u'down+peering'}], u'degraded_ratio': u'21.058', u'bytes_avail': 1809059700736, u'unfound_total': 18210}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:17,514.514 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:18,498.498 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:23,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:24,431.431 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:26,080.080 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:26,889.889 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:30:33.251449', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074952, u'health': u'HEALTH_OK', u'kb_used': 11254032, u'store_stats': {u'bytes_total': 4988542, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:30:33.969024', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436511824, u'health': u'HEALTH_OK', u'kb_used': 11817160, u'store_stats': {u'bytes_total': 6037120, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:30:33.513510', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437074936, u'health': u'HEALTH_OK', u'kb_used': 11254048, u'store_stats': {u'bytes_total': 6036983, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 13455/56146 objects degraded (23.964%); 58/18731 unfound (0.310%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 15047793, u'degraded_objects': 13455, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 78349603032, u'degraded_total': 56146, u'bytes_used': 59052244992, u'unfound_ratio': u'0.310', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 1057107471, u'op_per_sec': 333, u'unfound_objects': 58, u'version': 1653, u'pgs_by_state': [{u'count': 4, u'state_name': u'active'}, {u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 133, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}, {u'count': 3, u'state_name': u'active+clean+replay'}], u'degraded_ratio': u'23.964', u'bytes_avail': 1809358045184, u'unfound_total': 18731}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:26,890.890 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:27,719.719 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:32,348.348 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:33,029.029 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:34,768.768 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:35,536.536 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'43 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 13322/56092 objects degraded (23.750%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 10307811, u'degraded_objects': 13322, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 78274105560, u'degraded_total': 56092, u'bytes_used': 59249623040, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1658, u'pgs_by_state': [{u'count': 2, u'state_name': u'active'}, {u'count': 43, u'state_name': u'active+recovery_wait'}, {u'count': 134, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}, {u'count': 3, u'state_name': u'active+clean+replay'}], u'degraded_ratio': u'23.750', u'bytes_avail': 1809160667136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:35,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:36,423.423 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:41,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:41,817.817 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:43,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:44,399.399 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'43 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12912/56047 objects degraded (23.038%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12226708, u'degraded_objects': 12912, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 78211191000, u'degraded_total': 56047, u'bytes_used': 59424411648, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1664, u'pgs_by_state': [{u'count': 2, u'state_name': u'active'}, {u'count': 43, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}, {u'count': 3, u'state_name': u'active+clean+replay'}], u'degraded_ratio': u'23.038', u'bytes_avail': 1808985878528}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:44,400.400 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:45,375.375 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:49,499.499 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:50,343.343 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:51,903.903 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:31:52,697.697 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'42 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12825/56011 objects degraded (22.897%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12529769, u'degraded_objects': 12825, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 78160859352, u'degraded_total': 56011, u'bytes_used': 58927878144, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1669, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 42, u'state_name': u'active+recovery_wait'}, {u'count': 137, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}, {u'count': 1, u'state_name': u'active+clean+replay'}], u'degraded_ratio': u'22.897', u'bytes_avail': 1809482412032}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:31:52,698.698 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:53,511.511 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:31:58,353.353 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:31:59,250.250 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:00,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:04,602.602 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'42 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12610/55915 objects degraded (22.552%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18000670, u'degraded_objects': 12610, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 78026641624, u'degraded_total': 55915, u'bytes_used': 59129028608, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1677, u'pgs_by_state': [{u'count': 1, u'state_name': u'active'}, {u'count': 42, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.552', u'bytes_avail': 1809281261568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:04,603.603 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:05,727.727 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:10,340.340 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:11,241.241 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:12,987.987 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:13,994.994 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'42 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12128/55843 objects degraded (21.718%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18072571, u'degraded_objects': 12128, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77925978328, u'degraded_total': 55843, u'bytes_used': 59279233024, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 1, u'version': 1683, u'pgs_by_state': [{u'count': 42, u'state_name': u'active+recovery_wait'}, {u'count': 140, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.718', u'bytes_avail': 1809131057152}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:13,995.995 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:14,811.811 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:19,308.308 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:19,867.867 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:21,580.580 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:22,371.371 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'17 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 12034/55783 objects degraded (21.573%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18271813, u'degraded_objects': 12034, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77842092248, u'degraded_total': 55783, u'bytes_used': 59062464512, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1687, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 140, u'state_name': u'active+clean'}, {u'count': 17, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.573', u'bytes_avail': 1809347825664}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:22,372.372 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:23,210.210 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:27,541.541 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:28,516.516 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:30,086.086 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:30,951.951 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:31:33.251738', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011264, u'health': u'HEALTH_OK', u'kb_used': 11317720, u'store_stats': {u'bytes_total': 7085694, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:31:33.969325', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436466488, u'health': u'HEALTH_OK', u'kb_used': 11862496, u'store_stats': {u'bytes_total': 9182848, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:31:33.513799', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437011064, u'health': u'HEALTH_OK', u'kb_used': 11317920, u'store_stats': {u'bytes_total': 9182711, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11892/55693 objects degraded (21.353%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18211534, u'degraded_objects': 11892, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77716263128, u'degraded_total': 55693, u'bytes_used': 59250450432, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1693, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 16, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.353', u'bytes_avail': 1809159839744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:30,952.952 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:31,704.704 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:32:31.554986 7fe28e7fc700 0 monclient: hunting for new mon 2014-04-01 16:32:31,774.774 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:36,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:36,703.703 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:38,476.476 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:39,388.388 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11776/55624 objects degraded (21.171%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 25394250, u'degraded_objects': 11776, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77619794136, u'degraded_total': 55624, u'bytes_used': 59379945472, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1698, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.171', u'bytes_avail': 1809030344704}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:39,389.389 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:40,207.207 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:44,538.538 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:45,486.486 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:46,861.861 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:50,675.675 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11631/55540 objects degraded (20.942%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12760241, u'degraded_objects': 11631, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77502353624, u'degraded_total': 55540, u'bytes_used': 59575357440, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1704, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.942', u'bytes_avail': 1808834932736}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:50,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:51,679.679 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:56,195.195 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:32:56,999.999 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:32:58,960.960 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:32:59,649.649 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'31 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11484/55450 objects degraded (20.711%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 15191542, u'degraded_objects': 11484, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77376524504, u'degraded_total': 55450, u'bytes_used': 59747733504, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1710, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.711', u'bytes_avail': 1808662556672}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:32:59,650.650 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:00,365.365 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:04,812.812 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:05,757.757 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:06,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:07,650.650 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11366/55384 objects degraded (20.522%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 15688604, u'degraded_objects': 11366, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77284249816, u'degraded_total': 55384, u'bytes_used': 59866222592, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1715, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.522', u'bytes_avail': 1808544067584}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:07,652.652 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:08,472.472 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:13,207.207 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:14,309.309 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:15,758.758 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:16,615.615 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'15 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11261/55324 objects degraded (20.355%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 8749087, u'degraded_objects': 11261, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77200363736, u'degraded_total': 55324, u'bytes_used': 60004216832, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1720, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 15, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.355', u'bytes_avail': 1808406073344}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:16,616.616 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:17,458.458 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:21,812.812 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:22,410.410 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:24,108.108 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:24,862.862 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 11150/55264 objects degraded (20.176%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 15072884, u'degraded_objects': 11150, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77116477656, u'degraded_total': 55264, u'bytes_used': 59937042432, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1725, u'pgs_by_state': [{u'count': 41, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.176', u'bytes_avail': 1808473247744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:24,863.863 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:25,636.636 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:29,765.765 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:30,593.593 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:32,236.236 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:33,048.048 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:32:33.252007', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945392, u'health': u'HEALTH_OK', u'kb_used': 11383592, u'store_stats': {u'bytes_total': 9182846, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891374}}, {u'last_updated': u'2014-04-01 07:32:33.969647', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436435048, u'health': u'HEALTH_OK', u'kb_used': 11893936, u'store_stats': {u'bytes_total': 12328576, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891376}}, {u'last_updated': u'2014-04-01 07:32:33.514012', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436945020, u'health': u'HEALTH_OK', u'kb_used': 11383964, u'store_stats': {u'bytes_total': 12328439, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2891239}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 10465/55201 objects degraded (18.958%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12570713, u'degraded_objects': 10465, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 77028397272, u'degraded_total': 55201, u'bytes_used': 59614179328, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1730, u'pgs_by_state': [{u'count': 39, u'state_name': u'active+recovery_wait'}, {u'count': 148, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'18.958', u'bytes_avail': 1808796110848}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:33,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:33,971.971 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:38,209.209 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:38,832.832 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:40,084.084 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:40,812.812 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'37 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 10074/55117 objects degraded (18.277%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 20411195, u'degraded_objects': 10074, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76910956760, u'degraded_total': 55117, u'bytes_used': 59009019904, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1735, u'pgs_by_state': [{u'count': 37, u'state_name': u'active+recovery_wait'}, {u'count': 149, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'18.277', u'bytes_avail': 1809401270272}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:40,813.813 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:41,617.617 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:45,690.690 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:46,254.254 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:47,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:48,373.373 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9683/55057 objects degraded (17.587%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 10863595, u'degraded_objects': 9683, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76827070680, u'degraded_total': 55057, u'bytes_used': 58092171264, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 0, u'version': 1740, u'pgs_by_state': [{u'count': 36, u'state_name': u'active+recovery_wait'}, {u'count': 150, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.587', u'bytes_avail': 1810318118912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:48,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:49,077.077 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:53,182.182 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:53,896.896 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:33:54,986.986 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:33:55,683.683 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:33:55.605648 7f82a82f0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a0009af0 sd=11 :59816 s=2 pgs=10 cs=1 l=1 c=0x7f82a0011310).injecting socket failure 2014-04-01 16:33:55,710.710 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'14 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'33 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9377/54988 objects degraded (17.053%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 38884754, u'degraded_objects': 9377, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76730601688, u'degraded_total': 54988, u'bytes_used': 58054402048, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1746, u'pgs_by_state': [{u'count': 33, u'state_name': u'active+recovery_wait'}, {u'count': 151, u'state_name': u'active+clean'}, {u'count': 14, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.053', u'bytes_avail': 1810355888128}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:33:55,711.711 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:33:56,531.531 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:00,444.444 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:01,169.169 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:02,404.404 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:03,126.126 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'33 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'41 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8547/54907 objects degraded (15.566%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12435287, u'degraded_objects': 8547, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76617355480, u'degraded_total': 54907, u'bytes_used': 57884766208, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1750, u'pgs_by_state': [{u'count': 33, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.566', u'bytes_avail': 1810525523968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:03,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:04,036.036 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:08,116.116 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:08,813.813 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:10,048.048 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:10,832.832 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'33 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8470/54814 objects degraded (15.452%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 10963420, u'degraded_objects': 8470, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76487332056, u'degraded_total': 54814, u'bytes_used': 57214365696, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1755, u'pgs_by_state': [{u'count': 33, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.452', u'bytes_avail': 1811195924480}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:10,833.833 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:11,591.591 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:15,534.534 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:16,272.272 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:17,358.358 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:18,128.128 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8349/54727 objects degraded (15.256%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 17273792, u'degraded_objects': 8349, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76365697240, u'degraded_total': 54727, u'bytes_used': 57292484608, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1758, u'pgs_by_state': [{u'count': 32, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.256', u'bytes_avail': 1811117805568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:18,130.130 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:18,958.958 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:23,024.024 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:23,586.586 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:24,845.845 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:25,570.570 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 8265/54661 objects degraded (15.120%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18803388, u'degraded_objects': 8265, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76273422552, u'degraded_total': 54661, u'bytes_used': 57015455744, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1762, u'pgs_by_state': [{u'count': 32, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.120', u'bytes_avail': 1811394834432}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:25,571.571 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:26,268.268 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:30,348.348 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:31,079.079 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:32,177.177 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:33,065.065 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:33:33.252216', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894788, u'health': u'HEALTH_OK', u'kb_used': 11434196, u'store_stats': {u'bytes_total': 4399977, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351385}}, {u'last_updated': u'2014-04-01 07:33:33.969928', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436413432, u'health': u'HEALTH_OK', u'kb_used': 11915552, u'store_stats': {u'bytes_total': 5448861, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351693}}, {u'last_updated': u'2014-04-01 07:33:33.543987', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436894592, u'health': u'HEALTH_OK', u'kb_used': 11434392, u'store_stats': {u'bytes_total': 4399989, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3351397}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7888/54562 objects degraded (14.457%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 2921934, u'degraded_objects': 7888, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 76135010520, u'degraded_total': 54562, u'bytes_used': 57009909760, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1766, u'pgs_by_state': [{u'count': 32, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.457', u'bytes_avail': 1811400380416}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:33,066.066 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:33,830.830 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:37,909.909 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:38,501.501 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:39,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:40,588.588 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7789/54445 objects degraded (14.306%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 17284516, u'degraded_objects': 7789, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75971432664, u'degraded_total': 54445, u'bytes_used': 56493617152, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1770, u'pgs_by_state': [{u'count': 32, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'14.306', u'bytes_avail': 1811916673024}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:40,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:41,302.302 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:45,383.383 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:46,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:47,187.187 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:47,920.920 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'31 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'37 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7538/54352 objects degraded (13.869%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 4813299, u'degraded_objects': 7538, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75841409240, u'degraded_total': 54352, u'bytes_used': 56352763904, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1775, u'pgs_by_state': [{u'count': 31, u'state_name': u'active+recovery_wait'}, {u'count': 157, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.869', u'bytes_avail': 1812057526272}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:47,921.921 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:48,723.723 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:52,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:53,357.357 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:54,700.700 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:34:55,360.360 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'37 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7433/54232 objects degraded (13.706%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 10311166, u'degraded_objects': 7433, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75673637080, u'degraded_total': 54232, u'bytes_used': 56138309632, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1779, u'pgs_by_state': [{u'count': 30, u'state_name': u'active+recovery_wait'}, {u'count': 157, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.706', u'bytes_avail': 1812271980544}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:34:55,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:34:56,057.057 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:34:57,739.739 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:34:57.657421 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a0009af0 sd=11 :59878 s=2 pgs=11 cs=1 l=1 c=0x7f82a000e260).injecting socket failure 2014-04-01 16:34:57,740.740 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:34:57.659701 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a0016490 sd=12 :59929 s=2 pgs=12 cs=1 l=1 c=0x7f82a0012240).injecting socket failure 2014-04-01 16:35:00,279.279 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:01,107.107 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:02,187.187 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:02,977.977 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 7120/54175 objects degraded (13.143%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7577406, u'degraded_objects': 7120, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75593945304, u'degraded_total': 54175, u'bytes_used': 55981592576, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 3, u'version': 1783, u'pgs_by_state': [{u'count': 30, u'state_name': u'active+recovery_wait'}, {u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.143', u'bytes_avail': 1812428697600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:02,978.978 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:03,742.742 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:07,659.659 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:08,408.408 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:09,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:10,271.271 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'40 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6991/54028 objects degraded (12.940%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 10410548, u'degraded_objects': 6991, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75388424408, u'degraded_total': 54028, u'bytes_used': 56055603200, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1788, u'pgs_by_state': [{u'count': 30, u'state_name': u'active+recovery_wait'}, {u'count': 158, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.940', u'bytes_avail': 1812354686976}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:10,273.273 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:11,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:14,974.974 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:15,684.684 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:16,764.764 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:17,594.594 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6904/53908 objects degraded (12.807%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 9875349, u'degraded_objects': 6904, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75220652248, u'degraded_total': 53908, u'bytes_used': 56136331264, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 2, u'version': 1793, u'pgs_by_state': [{u'count': 30, u'state_name': u'active+recovery_wait'}, {u'count': 159, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.807', u'bytes_avail': 1812273958912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:17,595.595 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:18,313.313 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:22,392.392 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:23,288.288 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:24,386.386 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:25,150.150 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'30 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6819/53812 objects degraded (12.672%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 20057194, u'degraded_objects': 6819, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 75086434520, u'degraded_total': 53812, u'bytes_used': 56223019008, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1797, u'pgs_by_state': [{u'count': 30, u'state_name': u'active+recovery_wait'}, {u'count': 159, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.672', u'bytes_avail': 1812187271168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:25,151.151 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:25,945.945 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:26,686.686 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:35:26.609177 7f82982e1700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6814/7801 pipe(0x7f82a0009af0 sd=11 :59930 s=2 pgs=13 cs=1 l=1 c=0x7f82a000e260).injecting socket failure 2014-04-01 16:35:30,027.027 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:30,808.808 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:32,462.462 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:33,250.250 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:34:33.252439', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843864, u'health': u'HEALTH_OK', u'kb_used': 11485120, u'store_stats': {u'bytes_total': 4189180, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:34:33.970202', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436383768, u'health': u'HEALTH_OK', u'kb_used': 11945216, u'store_stats': {u'bytes_total': 4713839, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:34:33.544224', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436843620, u'health': u'HEALTH_OK', u'kb_used': 11485364, u'store_stats': {u'bytes_total': 4713440, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'29 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6729/53683 objects degraded (12.535%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 2871827, u'degraded_objects': 6729, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 74906079448, u'degraded_total': 53683, u'bytes_used': 56315916288, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1802, u'pgs_by_state': [{u'count': 29, u'state_name': u'active+recovery_wait'}, {u'count': 159, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.535', u'bytes_avail': 1812094373888}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:33,252.252 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:33,937.937 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:35:33.843814 7f82981e0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6801/9819 pipe(0x7f82a00234d0 sd=9 :44621 s=2 pgs=2 cs=1 l=1 c=0x7f82a001e440).injecting socket failure 2014-04-01 16:35:34,184.184 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:38,421.421 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:38,988.988 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:40,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:41,261.261 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 22, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.005436', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.012560', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6666/53512 objects degraded (12.457%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 3715142, u'degraded_objects': 6666, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 74667004120, u'degraded_total': 53512, u'bytes_used': 56173023232, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1807, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.457', u'bytes_avail': 1812237266944}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:41,262.262 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:42,074.074 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:46,239.239 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:50,003.003 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:51,086.086 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:51,852.852 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6562/53329 objects degraded (12.305%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'degraded_objects': 6562, u'num_pgs': 198, u'data_bytes': 74411151576, u'degraded_total': 53329, u'bytes_used': 55394676736, u'op_per_sec': 2, u'write_bytes_sec': 0, u'version': 1815, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.305', u'bytes_avail': 1813015613440}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:51,853.853 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:52,667.667 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:56,580.580 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:35:57,195.195 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:35:58,611.611 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:35:59,399.399 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6430/53125 objects degraded (12.104%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6378781, u'degraded_objects': 6430, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 74125938904, u'degraded_total': 53125, u'bytes_used': 55453933568, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1820, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.104', u'bytes_avail': 1812956356608}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:35:59,400.400 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:00,114.114 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:04,194.194 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:04,972.972 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:06,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:06,777.777 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6375/53002 objects degraded (12.028%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 5170408, u'degraded_objects': 6375, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 73953972440, u'degraded_total': 53002, u'bytes_used': 55501369344, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1824, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.028', u'bytes_avail': 1812908920832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:06,778.778 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:07,689.689 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:11,779.779 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:12,504.504 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:13,756.756 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:14,334.334 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6249/52819 objects degraded (11.831%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12883221, u'degraded_objects': 6249, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 73698119896, u'degraded_total': 52819, u'bytes_used': 55575830528, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1829, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.831', u'bytes_avail': 1812834459648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:14,335.335 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:15,034.034 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:19,099.099 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:19,657.657 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:21,063.063 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:21,838.838 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6200/52714 objects degraded (11.762%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7806228, u'degraded_objects': 6200, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 73551319256, u'degraded_total': 52714, u'bytes_used': 55602929664, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1833, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.762', u'bytes_avail': 1812807360512}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:21,839.839 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:22,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:26,661.661 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:27,453.453 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:28,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:29,383.383 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:35:33.252699', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436790140, u'health': u'HEALTH_OK', u'kb_used': 11538844, u'store_stats': {u'bytes_total': 6810620, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:35:33.970420', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436352860, u'health': u'HEALTH_OK', u'kb_used': 11976124, u'store_stats': {u'bytes_total': 7859567, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:35:33.544556', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436789760, u'health': u'HEALTH_OK', u'kb_used': 11539224, u'store_stats': {u'bytes_total': 7859168, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 6073/52552 objects degraded (11.556%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18040272, u'degraded_objects': 6073, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 73324826840, u'degraded_total': 52552, u'bytes_used': 55695507456, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1838, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.556', u'bytes_avail': 1812714782720}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:29,384.384 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:30,448.448 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:35,018.018 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:35,937.937 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:37,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:38,278.278 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5971/52384 objects degraded (11.399%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 3896777, u'degraded_objects': 5971, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 73089945816, u'degraded_total': 52384, u'bytes_used': 55784165376, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1843, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.399', u'bytes_avail': 1812626124800}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:38,279.279 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:39,198.198 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:43,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:44,453.453 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:45,927.927 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:46,889.889 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5892/52207 objects degraded (11.286%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 5026479, u'degraded_objects': 5892, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 72842481880, u'degraded_total': 52207, u'bytes_used': 55833874432, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1848, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.286', u'bytes_avail': 1812576415744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:46,890.890 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:47,707.707 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:52,449.449 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:53,189.189 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:36:54,785.785 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:36:55,610.610 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'38 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5766/52033 objects degraded (11.081%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6410964, u'degraded_objects': 5766, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 72599212248, u'degraded_total': 52033, u'bytes_used': 55915077632, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1853, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 160, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.081', u'bytes_avail': 1812495212544}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:36:55,611.611 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:36:56,418.418 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:00,945.945 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:01,644.644 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:03,494.494 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:04,265.265 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'37 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5686/51820 objects degraded (10.973%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 18796237, u'degraded_objects': 5686, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 72301416664, u'degraded_total': 51820, u'bytes_used': 55987032064, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1859, u'pgs_by_state': [{u'count': 28, u'state_name': u'active+recovery_wait'}, {u'count': 161, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.973', u'bytes_avail': 1812423258112}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:04,266.266 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:05,124.124 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:09,754.754 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:10,550.550 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:12,302.302 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:13,103.103 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5435/51664 objects degraded (10.520%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 1945143, u'degraded_objects': 5435, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 72083312856, u'degraded_total': 51664, u'bytes_used': 55398240256, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 4, u'version': 1865, u'pgs_by_state': [{u'count': 27, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.520', u'bytes_avail': 1813012049920}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:13,104.104 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:13,920.920 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:18,777.777 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:19,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:20,656.656 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:21,413.413 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5349/51421 objects degraded (10.402%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7741447, u'degraded_objects': 5349, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 71743574232, u'degraded_total': 51421, u'bytes_used': 55401943040, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1870, u'pgs_by_state': [{u'count': 27, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.402', u'bytes_avail': 1813008347136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:21,414.414 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:21,557.557 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:37:21.479309 7f82987e6700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.37:6800/4936 pipe(0x7f82a001f1d0 sd=8 :35291 s=2 pgs=17 cs=1 l=1 c=0x7f82a001dcf0).injecting socket failure 2014-04-01 16:37:22,176.176 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:26,412.412 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:27,149.149 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:28,228.228 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:29,273.273 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:36:33.252905', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436739256, u'health': u'HEALTH_OK', u'kb_used': 11589728, u'store_stats': {u'bytes_total': 8907772, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664876}}, {u'last_updated': u'2014-04-01 07:36:33.970666', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436323280, u'health': u'HEALTH_OK', u'kb_used': 12005704, u'store_stats': {u'bytes_total': 11005295, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3665247}}, {u'last_updated': u'2014-04-01 07:36:33.549052', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436738720, u'health': u'HEALTH_OK', u'kb_used': 11590264, u'store_stats': {u'bytes_total': 11004896, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3664848}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5227/51256 objects degraded (10.198%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 2512346, u'degraded_objects': 5227, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 71512887512, u'degraded_total': 51256, u'bytes_used': 55460454400, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1875, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.198', u'bytes_avail': 1812949835776}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:29,274.274 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:30,244.244 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:35,046.046 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:35,760.760 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:36,855.855 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:37,587.587 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5179/51124 objects degraded (10.130%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 3577999, u'degraded_objects': 5179, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 71328338136, u'degraded_total': 51124, u'bytes_used': 55507099648, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1879, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.130', u'bytes_avail': 1812903190528}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:37,588.588 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:38,394.394 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:42,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:43,392.392 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:44,534.534 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:45,309.309 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 5055/50917 objects degraded (9.928%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 11310499, u'degraded_objects': 5055, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 71038931160, u'degraded_total': 50917, u'bytes_used': 55592833024, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1884, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.928', u'bytes_avail': 1812817457152}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:45,311.311 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:46,125.125 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:50,211.211 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:50,887.887 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:52,120.120 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:37:53,001.001 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4986/50755 objects degraded (9.824%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7696503, u'degraded_objects': 4986, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 70812438744, u'degraded_total': 50755, u'bytes_used': 55658307584, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1891, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.824', u'bytes_avail': 1812751982592}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:37:53,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:53,706.706 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:57,776.776 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:37:58,519.519 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:37:59,596.596 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:00,386.386 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4903/50584 objects degraded (9.693%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12884123, u'degraded_objects': 4903, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 70573363416, u'degraded_total': 50584, u'bytes_used': 55709188096, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1895, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.693', u'bytes_avail': 1812701102080}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:00,388.388 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:01,161.161 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:05,373.373 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:05,987.987 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:07,226.226 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:07,955.955 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4841/50458 objects degraded (9.594%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6293820, u'degraded_objects': 4841, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 70397202648, u'degraded_total': 50458, u'bytes_used': 55759544320, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1899, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.594', u'bytes_avail': 1812650745856}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:07,956.956 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:08,714.714 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:12,788.788 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:13,410.410 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:14,646.646 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:15,432.432 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4746/50257 objects degraded (9.443%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7521787, u'degraded_objects': 4746, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 70116184280, u'degraded_total': 50257, u'bytes_used': 55814680576, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1904, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.443', u'bytes_avail': 1812595609600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:15,433.433 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:16,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:20,267.267 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:21,021.021 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:22,117.117 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:22,828.828 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4668/50095 objects degraded (9.318%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6283989, u'degraded_objects': 4668, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 69889691864, u'degraded_total': 50095, u'bytes_used': 55881072640, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1909, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.318', u'bytes_avail': 1812529217536}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:22,829.829 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:23,522.522 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:27,449.449 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:28,241.241 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:29,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:30,052.052 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:37:33.253152', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436700132, u'health': u'HEALTH_OK', u'kb_used': 11628852, u'store_stats': {u'bytes_total': 5208238, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:37:33.970876', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436301056, u'health': u'HEALTH_OK', u'kb_used': 12027928, u'store_stats': {u'bytes_total': 5208380, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:37:33.566329', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436699744, u'health': u'HEALTH_OK', u'kb_used': 11629240, u'store_stats': {u'bytes_total': 5208279, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'36 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4579/49960 objects degraded (9.165%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 5024532, u'degraded_objects': 4579, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 69700948184, u'degraded_total': 49960, u'bytes_used': 55913799680, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1913, u'pgs_by_state': [{u'count': 26, u'state_name': u'active+recovery_wait'}, {u'count': 162, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.165', u'bytes_avail': 1812496490496}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:30,054.054 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:30,859.859 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:34,786.786 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:35,350.350 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:36,592.592 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:37,326.326 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'35 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4545/49825 objects degraded (9.122%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 3878927, u'degraded_objects': 4545, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 69512204504, u'degraded_total': 49825, u'bytes_used': 55979085824, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1917, u'pgs_by_state': [{u'count': 25, u'state_name': u'active+recovery_wait'}, {u'count': 163, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'9.122', u'bytes_avail': 1812431204352}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:37,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:38,089.089 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:42,168.168 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:43,047.047 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:44,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:44,916.916 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'35 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4441/49633 objects degraded (8.948%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12112107, u'degraded_objects': 4441, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 69243769048, u'degraded_total': 49633, u'bytes_used': 56025776128, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1922, u'pgs_by_state': [{u'count': 25, u'state_name': u'active+recovery_wait'}, {u'count': 163, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.948', u'bytes_avail': 1812384514048}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:44,917.917 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:45,678.678 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:49,754.754 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:50,747.747 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:51,856.856 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:38:52,627.627 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'35 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4386/49525 objects degraded (8.856%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 3680855, u'degraded_objects': 4386, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 69092774104, u'degraded_total': 49525, u'bytes_used': 56103006208, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 8, u'version': 1926, u'pgs_by_state': [{u'count': 25, u'state_name': u'active+recovery_wait'}, {u'count': 163, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.856', u'bytes_avail': 1812307283968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:38:52,628.628 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:38:53,332.332 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:38:57,250.250 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:00,974.974 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:02,057.057 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:02,843.843 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'32 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3930/49267 objects degraded (7.977%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 8534871, u'degraded_objects': 3930, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 68732063960, u'degraded_total': 49267, u'bytes_used': 55840776192, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1933, u'pgs_by_state': [{u'count': 21, u'state_name': u'active+recovery_wait'}, {u'count': 166, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.977', u'bytes_avail': 1812569513984}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:02,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:03,599.599 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:04,148.148 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:39:04.072189 7f82a82f0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6801/9819 pipe(0x7f82a00202a0 sd=9 :44936 s=2 pgs=5 cs=1 l=1 c=0x7f82a0022f00).injecting socket failure 2014-04-01 16:39:07,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:08,243.243 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:09,483.483 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:10,191.191 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'29 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3834/49111 objects degraded (7.807%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 4911292, u'degraded_objects': 3834, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 68513960152, u'degraded_total': 49111, u'bytes_used': 55411773440, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1938, u'pgs_by_state': [{u'count': 20, u'state_name': u'active+recovery_wait'}, {u'count': 169, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.807', u'bytes_avail': 1812998516736}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:10,192.192 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:10,952.952 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:14,865.865 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:15,589.589 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:16,677.677 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:17,481.481 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'29 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3772/48940 objects degraded (7.707%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 2403595, u'degraded_objects': 3772, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 68274884824, u'degraded_total': 48940, u'bytes_used': 55481462784, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1943, u'pgs_by_state': [{u'count': 19, u'state_name': u'active+recovery_wait'}, {u'count': 169, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.707', u'bytes_avail': 1812928827392}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:17,482.482 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:18,234.234 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:22,323.323 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:23,116.116 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:24,205.205 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:24,907.907 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3532/48742 objects degraded (7.246%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 9076581, u'degraded_objects': 3532, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 67998060760, u'degraded_total': 48742, u'bytes_used': 55547084800, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 12, u'version': 1947, u'pgs_by_state': [{u'count': 19, u'state_name': u'active+recovery_wait'}, {u'count': 170, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.246', u'bytes_avail': 1812863205376}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:24,908.908 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:25,677.677 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:29,760.760 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:30,627.627 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:32,078.078 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:32,879.879 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:38:33.256669', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436651188, u'health': u'HEALTH_OK', u'kb_used': 11677796, u'store_stats': {u'bytes_total': 7305390, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:38:33.971138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436272940, u'health': u'HEALTH_OK', u'kb_used': 12056044, u'store_stats': {u'bytes_total': 8354108, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:38:33.566563', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436650812, u'health': u'HEALTH_OK', u'kb_used': 11678172, u'store_stats': {u'bytes_total': 8354007, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'28 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3498/48595 objects degraded (7.198%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 1251124, u'degraded_objects': 3498, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 67792539864, u'degraded_total': 48595, u'bytes_used': 55070789632, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 0, u'op_per_sec': 7, u'version': 1951, u'pgs_by_state': [{u'count': 19, u'state_name': u'active+recovery_wait'}, {u'count': 170, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.198', u'bytes_avail': 1813339500544}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:32,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:33,704.704 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:39,171.171 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:39,865.865 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:40,960.960 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:41,735.735 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3288/48352 objects degraded (6.800%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6312235, u'degraded_objects': 3288, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 67452801240, u'degraded_total': 48352, u'bytes_used': 54858854400, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 9, u'version': 1957, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 171, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.800', u'bytes_avail': 1813551435776}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:41,736.736 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:42,433.433 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:46,502.502 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:47,122.122 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:48,530.530 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:49,299.299 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'27 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3185/48151 objects degraded (6.615%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 22918557, u'degraded_objects': 3185, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 67171782872, u'degraded_total': 48151, u'bytes_used': 54656409600, u'recovering_objects_per_sec': 5, u'write_bytes_sec': 0, u'op_per_sec': 17, u'version': 1963, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 171, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.615', u'bytes_avail': 1813753880576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:49,300.300 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:50,241.241 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:54,518.518 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:55,143.143 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:39:56,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:39:57,290.290 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'18 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'26 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3047/48061 objects degraded (6.340%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 9105028, u'degraded_objects': 3047, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 67045953752, u'degraded_total': 48061, u'bytes_used': 54528200704, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 6, u'version': 1967, u'pgs_by_state': [{u'count': 18, u'state_name': u'active+recovery_wait'}, {u'count': 172, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.340', u'bytes_avail': 1813882089472}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:39:57,291.291 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:39:58,079.079 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:02,212.212 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:02,985.985 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:04,071.071 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:04,850.850 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2689/47854 objects degraded (5.619%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 12836608, u'degraded_objects': 2689, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 66756546776, u'degraded_total': 47854, u'bytes_used': 53734944768, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 0, u'op_per_sec': 11, u'version': 1972, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 173, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.619', u'bytes_avail': 1814675345408}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:04,851.851 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:05,620.620 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:09,533.533 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:10,306.306 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:11,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:12,208.208 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2519/47734 objects degraded (5.277%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 8821387, u'degraded_objects': 2519, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 66588774616, u'degraded_total': 47734, u'bytes_used': 53768908800, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 5, u'version': 1976, u'pgs_by_state': [{u'count': 16, u'state_name': u'active+recovery_wait'}, {u'count': 174, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'5.277', u'bytes_avail': 1814641381376}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:12,210.210 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:12,907.907 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:16,978.978 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:17,695.695 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:18,783.783 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:19,557.557 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'21 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1845/47461 objects degraded (3.887%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 7521018, u'degraded_objects': 1845, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 66207092952, u'degraded_total': 47461, u'bytes_used': 53694038016, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 15, u'version': 1981, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 177, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.887', u'bytes_avail': 1814716252160}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:19,558.558 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:20,323.323 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:24,395.395 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:25,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:26,284.284 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:27,074.074 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:39:33.256953', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598696, u'health': u'HEALTH_OK', u'kb_used': 11730288, u'store_stats': {u'bytes_total': 8353966, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111070}}, {u'last_updated': u'2014-04-01 07:39:33.974750', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436242608, u'health': u'HEALTH_OK', u'kb_used': 12086376, u'store_stats': {u'bytes_total': 11499836, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111212}}, {u'last_updated': u'2014-04-01 07:39:33.566931', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436598376, u'health': u'HEALTH_OK', u'kb_used': 11730608, u'store_stats': {u'bytes_total': 11499735, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3111111}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'20 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1903/47311 objects degraded (4.022%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 6314309, u'degraded_objects': 1903, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 65997377752, u'degraded_total': 47311, u'bytes_used': 53484859392, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 0, u'op_per_sec': 10, u'version': 1985, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 178, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.022', u'bytes_avail': 1814925430784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:27,074.074 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:27,702.702 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 07:40:27.626073 7f82a82f0700 0 -- 10.214.133.37:0/1005903 >> 10.214.133.38:6801/9819 pipe(0x7f82a001f070 sd=11 :45014 s=2 pgs=6 cs=1 l=1 c=0x7f82a00013b0).injecting socket failure 2014-04-01 16:40:27,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:31,955.955 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:32,699.699 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:33,795.795 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:34,707.707 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1483/47065 objects degraded (3.151%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 8683551, u'degraded_objects': 1483, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 65653444824, u'degraded_total': 47065, u'bytes_used': 53257371648, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 12, u'version': 1990, u'pgs_by_state': [{u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 184, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.151', u'bytes_avail': 1815152918528}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:34,708.708 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:35,414.414 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:39,485.485 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:40,200.200 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:41,282.282 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:42,001.001 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 24, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.004478', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.010160', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1402/46924 objects degraded (2.988%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'degraded_objects': 1402, u'num_pgs': 198, u'data_bytes': 65456312536, u'degraded_total': 46924, u'bytes_used': 52898136064, u'op_per_sec': 11, u'write_bytes_sec': 0, u'version': 1994, u'pgs_by_state': [{u'count': 1, u'state_name': u'active+recovery_wait'}, {u'count': 187, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.988', u'bytes_avail': 1815512154112}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:42,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:42,606.606 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 07:40:42.530680 7ff2837fe700 0 monclient: hunting for new mon 2014-04-01 16:40:42,757.757 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:46,835.835 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:47,389.389 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:48,641.641 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:49,425.425 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 26, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003725', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.008252', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery -11/46651 objects degraded (-0.024%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 76 > pgp_num 56'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'recovering_bytes_per_sec': 8417257, u'degraded_objects': 18446744073709551605L, u'num_pgs': 198, u'recovering_keys_per_sec': 0, u'data_bytes': 65074630872, u'degraded_total': 46651, u'bytes_used': 52158054400, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 0, u'op_per_sec': 15, u'version': 1999, u'pgs_by_state': [{u'count': 195, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'-0.024', u'bytes_avail': 1816252235776}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 133, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:49,426.426 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:50,205.205 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:53,382.382 INFO:teuthology.task.thrashosds.ceph_manager:removing pool_name unique_pool_0 2014-04-01 16:40:53,383.383 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados rmpool unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it' 2014-04-01 16:40:54,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:54,998.998 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:40:55,177.177 INFO:teuthology.orchestra.run.out:[10.214.133.37]: successfully deleted pool unique_pool_0 2014-04-01 16:40:55,178.178 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2014-04-01 16:40:55,178.178 INFO:teuthology.task.thrashosds:joining thrashosds 2014-04-01 16:40:56,080.080 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:40:56,808.808 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 26, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003725', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.008252', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'num_pgs': 122, u'data_bytes': 4288, u'bytes_used': 51306168320, u'version': 2003, u'pgs_by_state': [{u'count': 122, u'state_name': u'active+clean'}], u'bytes_avail': 1817104121856}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 134, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:40:56,809.809 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:40:57,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:01,205.205 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:41:01,834.834 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:02,605.605 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:41:03,404.404 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 26, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003725', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.008252', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'num_pgs': 122, u'data_bytes': 4288, u'bytes_used': 45357146112, u'version': 2007, u'pgs_by_state': [{u'count': 122, u'state_name': u'active+clean'}], u'bytes_avail': 1823053144064}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 134, u'num_in_osds': 4}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:41:03,405.405 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 16:41:03,405.405 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 16:41:03,405.405 INFO:teuthology.task.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.714896 2014-04-01 16:41:03,406.406 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 1 0.714895672807' 2014-04-01 16:41:04,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.1 primary-affinity to 0.714896 (8468512) 2014-04-01 16:41:09,406.406 INFO:teuthology.task.thrashosds.thrasher:inning osd 2014-04-01 16:41:09,407.407 INFO:teuthology.task.thrashosds.thrasher:Adding osd 5 2014-04-01 16:41:09,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-04-01 16:41:11,398.398 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.5. 2014-04-01 16:41:11,399.399 INFO:teuthology.task.thrashosds.thrasher:Added osd 5 2014-04-01 16:41:11,399.399 INFO:teuthology.task.thrashosds.thrasher:inning osd 2014-04-01 16:41:11,400.400 INFO:teuthology.task.thrashosds.thrasher:Adding osd 4 2014-04-01 16:41:11,400.400 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 4' 2014-04-01 16:41:12,342.342 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.4. 2014-04-01 16:41:12,354.354 INFO:teuthology.task.thrashosds.thrasher:Added osd 4 2014-04-01 16:41:12,354.354 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 16:41:12,355.355 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:41:13,196.196 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:13,824.824 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:41:14,542.542 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:15,168.168 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:41:15,890.890 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 26, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003725', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.008252', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 1968498966528, u'num_pgs': 122, u'data_bytes': 4288, u'bytes_used': 4401930240, u'version': 2015, u'pgs_by_state': [{u'count': 7, u'state_name': u'peering'}, {u'count': 115, u'state_name': u'active+clean'}], u'bytes_avail': 1864008359936}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 139, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:41:15,891.891 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:41:16,583.583 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:20,198.198 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 16:41:20,972.972 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:21,613.613 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 16:41:22,338.338 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 06:39:22.238827', u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593', u'created': u'2014-04-01 06:39:22.238827'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 26, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.003725', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.008252', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 07:40:33.257299', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568392, u'health': u'HEALTH_OK', u'kb_used': 11760592, u'store_stats': {u'bytes_total': 4461590, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3412998}}, {u'last_updated': u'2014-04-01 07:40:33.974995', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436215440, u'health': u'HEALTH_OK', u'kb_used': 12113544, u'store_stats': {u'bytes_total': 5510629, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413461}}, {u'last_updated': u'2014-04-01 07:40:33.567164', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436568068, u'health': u'HEALTH_OK', u'kb_used': 11760916, u'store_stats': {u'bytes_total': 5510295, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3413127}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 54 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 15, u'num_pgs': 122, u'recovering_keys_per_sec': 2, u'data_bytes': 4288, u'bytes_used': 1113554944, u'recovering_objects_per_sec': 0, u'version': 2017, u'pgs_by_state': [{u'count': 122, u'state_name': u'active+clean'}], u'bytes_avail': 2801501880320}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 139, u'num_in_osds': 6}}, u'fsid': u'a7571e41-653a-4300-8fe0-63e1a9732593'} 2014-04-01 16:41:22,339.339 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 16:41:22,340.340 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2014-04-01 16:41:22,340.340 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:41:23,034.034 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:24,641.641 INFO:teuthology.task.ceph:Scrubbing osd osd.0 2014-04-01 16:41:24,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.0' 2014-04-01 16:41:25,276.276 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.0 instructed to scrub 2014-04-01 16:41:25,289.289 INFO:teuthology.task.ceph:Scrubbing osd osd.1 2014-04-01 16:41:25,290.290 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.1' 2014-04-01 16:41:25,986.986 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.1 instructed to scrub 2014-04-01 16:41:25,999.999 INFO:teuthology.task.ceph:Scrubbing osd osd.2 2014-04-01 16:41:25,999.999 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.2' 2014-04-01 16:41:26,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.2 instructed to scrub 2014-04-01 16:41:26,750.750 INFO:teuthology.task.ceph:Scrubbing osd osd.3 2014-04-01 16:41:26,750.750 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.3' 2014-04-01 16:41:27,432.432 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.3 instructed to scrub 2014-04-01 16:41:27,445.445 INFO:teuthology.task.ceph:Scrubbing osd osd.4 2014-04-01 16:41:27,445.445 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.4' 2014-04-01 16:41:28,185.185 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.4 instructed to scrub 2014-04-01 16:41:28,198.198 INFO:teuthology.task.ceph:Scrubbing osd osd.5 2014-04-01 16:41:28,199.199 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd scrub osd.5' 2014-04-01 16:41:28,981.981 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osd.5 instructed to scrub 2014-04-01 16:41:28,981.981 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:41:30,049.049 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:30,872.872 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:41:40,846.846 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:41:41,621.621 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:42,254.254 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:41:52,237.237 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:41:52,955.955 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:41:53,583.583 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:42:03,568.568 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:42:04,521.521 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:42:05,149.149 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:42:15,132.132 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:42:15,838.838 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:42:16,465.465 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:42:26,447.447 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:42:27,227.227 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:42:27,866.866 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:42:37,849.849 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:42:38,474.474 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:42:39,264.264 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:42:49,240.240 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:42:49,952.952 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:42:50,581.581 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:43:00,564.564 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:43:01,341.341 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:43:01,981.981 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:43:11,965.965 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:43:12,730.730 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:43:13,357.357 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:43:23,342.342 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:43:24,047.047 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:43:24,673.673 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:43:34,657.657 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:43:35,400.400 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:43:36,072.072 INFO:teuthology.task.ceph:Still waiting for all pgs to be scrubbed. 2014-04-01 16:43:46,050.050 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-04-01 16:43:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 16:43:50,402.402 INFO:teuthology.task.ceph:Exiting scrub checking -- not all pgs scrubbed. 2014-04-01 16:43:50,402.402 INFO:teuthology.misc:Shutting down mds daemons... 2014-04-01 16:43:50,402.402 DEBUG:teuthology.task.ceph.mds.a:waiting for process to exit 2014-04-01 16:43:50,562.562 INFO:teuthology.task.ceph.mds.a:Stopped 2014-04-01 16:43:50,562.562 INFO:teuthology.misc:Shutting down osd daemons... 2014-04-01 16:43:50,563.563 DEBUG:teuthology.task.ceph.osd.1:waiting for process to exit 2014-04-01 16:43:50,748.748 INFO:teuthology.task.ceph.osd.1:Stopped 2014-04-01 16:43:50,749.749 DEBUG:teuthology.task.ceph.osd.0:waiting for process to exit 2014-04-01 16:43:51,107.107 INFO:teuthology.task.ceph.osd.0:Stopped 2014-04-01 16:43:51,108.108 DEBUG:teuthology.task.ceph.osd.3:waiting for process to exit 2014-04-01 16:43:51,279.279 INFO:teuthology.task.ceph.osd.3:Stopped 2014-04-01 16:43:51,279.279 DEBUG:teuthology.task.ceph.osd.2:waiting for process to exit 2014-04-01 16:43:51,470.470 INFO:teuthology.task.ceph.osd.2:Stopped 2014-04-01 16:43:51,471.471 DEBUG:teuthology.task.ceph.osd.5:waiting for process to exit 2014-04-01 16:43:51,631.631 INFO:teuthology.task.ceph.osd.5:Stopped 2014-04-01 16:43:51,632.632 DEBUG:teuthology.task.ceph.osd.4:waiting for process to exit 2014-04-01 16:43:51,983.983 INFO:teuthology.task.ceph.osd.4:Stopped 2014-04-01 16:43:51,983.983 INFO:teuthology.misc:Shutting down mon daemons... 2014-04-01 16:43:51,983.983 DEBUG:teuthology.task.ceph.mon.a:waiting for process to exit 2014-04-01 16:43:52,148.148 INFO:teuthology.task.ceph.mon.a:Stopped 2014-04-01 16:43:52,149.149 DEBUG:teuthology.task.ceph.mon.c:waiting for process to exit 2014-04-01 16:43:52,496.496 INFO:teuthology.task.ceph.mon.c:Stopped 2014-04-01 16:43:52,497.497 DEBUG:teuthology.task.ceph.mon.b:waiting for process to exit 2014-04-01 16:43:52,657.657 INFO:teuthology.task.ceph.mon.b:Stopped 2014-04-01 16:43:52,657.657 INFO:teuthology.task.ceph:Checking cluster log for badness... 2014-04-01 16:43:52,657.657 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo egrep '\\[ERR\\]|\\[WRN\\]|\\[SEC\\]' /var/log/ceph/ceph.log | egrep -v 'wrongly marked me down' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | head -n 1" 2014-04-01 16:43:53,319.319 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 16:43:53,320.320 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-0' 2014-04-01 16:43:54,094.094 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 16:43:54,095.095 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-1' 2014-04-01 16:43:54,711.711 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 16:43:54,712.712 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-2' 2014-04-01 16:43:55,385.385 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 16:43:55,386.386 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-3' 2014-04-01 16:43:56,004.004 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 16:43:56,005.005 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-4' 2014-04-01 16:43:56,596.596 INFO:teuthology.task.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 16:43:56,596.596 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-5' 2014-04-01 16:43:57,080.080 INFO:teuthology.task.ceph:Cleaning ceph cluster... 2014-04-01 16:43:57,081.081 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/data /home/ubuntu/cephtest/monmap' 2014-04-01 16:43:57,466.466 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/data /home/ubuntu/cephtest/monmap' 2014-04-01 16:43:57,934.934 INFO:teuthology.task.ceph:Checking for errors in any valgrind logs... 2014-04-01 16:43:57,935.935 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 2014-04-01 16:43:58,393.393 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 2014-04-01 16:43:58,543.543 INFO:teuthology.orchestra.run.err:[10.214.133.37]: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2014-04-01 16:43:58,895.895 INFO:teuthology.orchestra.run.err:[10.214.133.38]: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2014-04-01 16:43:58,905.905 INFO:teuthology.task.ceph:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/kcon_most... 2014-04-01 16:43:58,905.905 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/kcon_most' 2014-04-01 16:43:59,396.396 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/kcon_most' 2014-04-01 16:43:59,739.739 DEBUG:teuthology.run_tasks:Unwinding manager install 2014-04-01 16:43:59,740.740 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 16:44:00,343.343 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 16:44:00,344.344 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 16:44:00,345.345 INFO:teuthology.task.install:Removing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg on Debian system. 2014-04-01 16:44:00,345.345 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'for d in ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done' 2014-04-01 16:44:00,809.809 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 16:44:00,810.810 INFO:teuthology.task.install:Removing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg on Debian system. 2014-04-01 16:44:00,811.811 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'for d in ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done' 2014-04-01 16:44:01,149.149 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:01,646.646 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:02,274.274 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:02,291.291 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:02,618.618 INFO:teuthology.orchestra.run.out:[10.214.133.37]: The following packages will be REMOVED: 2014-04-01 16:44:02,618.618 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph* ceph-dbg* ceph-mds* ceph-mds-dbg* 2014-04-01 16:44:02,874.874 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:02,892.892 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:03,233.233 INFO:teuthology.orchestra.run.out:[10.214.133.38]: The following packages will be REMOVED: 2014-04-01 16:44:03,234.234 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph* ceph-dbg* ceph-mds* ceph-mds-dbg* 2014-04-01 16:44:03,695.695 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 4 to remove and 94 not upgraded. 2014-04-01 16:44:03,695.695 INFO:teuthology.orchestra.run.out:[10.214.133.37]: After this operation, 572 MB disk space will be freed. 2014-04-01 16:44:04,241.241 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 4 to remove and 94 not upgraded. 2014-04-01 16:44:04,242.242 INFO:teuthology.orchestra.run.out:[10.214.133.38]: After this operation, 572 MB disk space will be freed. 2014-04-01 16:44:05,612.612 INFO:teuthology.orchestra.run.out:[10.214.133.37]: (Reading database ... 81155 files and directories currently installed.) 2014-04-01 16:44:05,615.615 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-mds-dbg ... 2014-04-01 16:44:06,136.136 INFO:teuthology.orchestra.run.out:[10.214.133.38]: (Reading database ... 81160 files and directories currently installed.) 2014-04-01 16:44:06,137.137 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-mds-dbg ... 2014-04-01 16:44:06,216.216 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-mds ... 2014-04-01 16:44:06,312.312 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mds-all stop/waiting 2014-04-01 16:44:06,533.533 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /etc/init.d/ceph: ceph conf /etc/ceph/ceph.conf not found; system is not configured. 2014-04-01 16:44:06,725.725 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-mds ... 2014-04-01 16:44:06,872.872 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-mds-all stop/waiting 2014-04-01 16:44:06,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Purging configuration files for ceph-mds ... 2014-04-01 16:44:07,087.087 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /etc/init.d/ceph: ceph conf /etc/ceph/ceph.conf not found; system is not configured. 2014-04-01 16:44:07,414.414 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-dbg ... 2014-04-01 16:44:07,444.444 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Purging configuration files for ceph-mds ... 2014-04-01 16:44:07,884.884 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph ... 2014-04-01 16:44:07,933.933 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-dbg ... 2014-04-01 16:44:07,944.944 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-all stop/waiting 2014-04-01 16:44:07,959.959 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /etc/init.d/ceph: ceph conf /etc/ceph/ceph.conf not found; system is not configured. 2014-04-01 16:44:08,393.393 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph ... 2014-04-01 16:44:08,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-all stop/waiting 2014-04-01 16:44:08,471.471 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /etc/init.d/ceph: ceph conf /etc/ceph/ceph.conf not found; system is not configured. 2014-04-01 16:44:08,879.879 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Purging configuration files for ceph ... 2014-04-01 16:44:09,380.380 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Purging configuration files for ceph ... 2014-04-01 16:44:10,072.072 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dpkg: warning: while removing ceph, directory '/var/lib/ceph/osd' not empty so not removed. 2014-04-01 16:44:10,072.072 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dpkg: warning: while removing ceph, directory '/var/lib/ceph/mon' not empty so not removed. 2014-04-01 16:44:10,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.38]: dpkg: warning: while removing ceph, directory '/var/lib/ceph/mds' not empty so not removed. 2014-04-01 16:44:10,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.38]: dpkg: warning: while removing ceph, directory '/var/lib/ceph/osd' not empty so not removed. 2014-04-01 16:44:10,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.38]: dpkg: warning: while removing ceph, directory '/var/lib/ceph/mon' not empty so not removed. 2014-04-01 16:44:10,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for ureadahead ... 2014-04-01 16:44:10,792.792 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for man-db ... 2014-04-01 16:44:10,830.830 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for ureadahead ... 2014-04-01 16:44:10,939.939 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for man-db ... 2014-04-01 16:44:12,160.160 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for libc-bin ... 2014-04-01 16:44:12,170.170 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for libc-bin ... 2014-04-01 16:44:12,204.204 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ldconfig deferred processing now taking place 2014-04-01 16:44:12,235.235 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ldconfig deferred processing now taking place 2014-04-01 16:44:13,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:13,765.765 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:14,063.063 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:14,066.066 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:14,130.130 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:14,131.131 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:14,332.332 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-dbg is not installed, so not removed 2014-04-01 16:44:14,333.333 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:14,345.345 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:14,398.398 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-dbg is not installed, so not removed 2014-04-01 16:44:14,399.399 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:14,413.413 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:14,699.699 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:14,702.702 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:14,766.766 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:14,768.768 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:14,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-mds is not installed, so not removed 2014-04-01 16:44:14,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:14,953.953 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:15,007.007 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-mds is not installed, so not removed 2014-04-01 16:44:15,007.007 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:15,016.016 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:15,203.203 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:15,204.204 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:15,265.265 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:15,266.266 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:15,388.388 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-mds-dbg is not installed, so not removed 2014-04-01 16:44:15,389.389 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:15,398.398 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:15,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-mds-dbg is not installed, so not removed 2014-04-01 16:44:15,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:15,464.464 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:15,745.745 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:15,746.746 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:15,807.807 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:15,808.808 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:16,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: The following packages will be REMOVED: 2014-04-01 16:44:16,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-common* ceph-common-dbg* ceph-test* ceph-test-dbg* radosgw* 2014-04-01 16:44:16,059.059 INFO:teuthology.orchestra.run.out:[10.214.133.38]: radosgw-dbg* 2014-04-01 16:44:16,119.119 INFO:teuthology.orchestra.run.out:[10.214.133.37]: The following packages will be REMOVED: 2014-04-01 16:44:16,120.120 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-common* ceph-common-dbg* ceph-test* ceph-test-dbg* radosgw* 2014-04-01 16:44:16,120.120 INFO:teuthology.orchestra.run.out:[10.214.133.37]: radosgw-dbg* 2014-04-01 16:44:16,204.204 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 6 to remove and 94 not upgraded. 2014-04-01 16:44:16,205.205 INFO:teuthology.orchestra.run.out:[10.214.133.38]: After this operation, 1,830 MB disk space will be freed. 2014-04-01 16:44:16,265.265 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 6 to remove and 94 not upgraded. 2014-04-01 16:44:16,265.265 INFO:teuthology.orchestra.run.out:[10.214.133.37]: After this operation, 1,830 MB disk space will be freed. 2014-04-01 16:44:16,312.312 INFO:teuthology.orchestra.run.out:[10.214.133.38]: (Reading database ... 80970 files and directories currently installed.) 2014-04-01 16:44:16,315.315 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing radosgw-dbg ... 2014-04-01 16:44:16,373.373 INFO:teuthology.orchestra.run.out:[10.214.133.37]: (Reading database ... 80965 files and directories currently installed.) 2014-04-01 16:44:16,375.375 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing radosgw-dbg ... 2014-04-01 16:44:16,776.776 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing radosgw ... 2014-04-01 16:44:16,829.829 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing radosgw ... 2014-04-01 16:44:16,845.845 INFO:teuthology.orchestra.run.err:[10.214.133.38]: stop: Unknown instance: 2014-04-01 16:44:16,878.878 INFO:teuthology.orchestra.run.out:[10.214.133.38]: No /usr/bin/radosgw found running; none killed. 2014-04-01 16:44:16,883.883 INFO:teuthology.orchestra.run.err:[10.214.133.37]: stop: Unknown instance: 2014-04-01 16:44:16,932.932 INFO:teuthology.orchestra.run.out:[10.214.133.37]: No /usr/bin/radosgw found running; none killed. 2014-04-01 16:44:18,091.091 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Purging configuration files for radosgw ... 2014-04-01 16:44:18,094.094 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Purging configuration files for radosgw ... 2014-04-01 16:44:18,095.095 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-test-dbg ... 2014-04-01 16:44:18,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dpkg: warning: while removing radosgw, directory '/var/lib/ceph' not empty so not removed. 2014-04-01 16:44:18,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.38]: dpkg: warning: while removing radosgw, directory '/var/lib/ceph' not empty so not removed. 2014-04-01 16:44:18,096.096 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-test-dbg ... 2014-04-01 16:44:18,348.348 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-test ... 2014-04-01 16:44:18,363.363 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-test ... 2014-04-01 16:44:18,787.787 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-common-dbg ... 2014-04-01 16:44:18,793.793 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-common-dbg ... 2014-04-01 16:44:19,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-common ... 2014-04-01 16:44:19,210.210 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-common ... 2014-04-01 16:44:19,459.459 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Purging configuration files for ceph-common ... 2014-04-01 16:44:19,469.469 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Purging configuration files for ceph-common ... 2014-04-01 16:44:19,796.796 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for ureadahead ... 2014-04-01 16:44:19,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for ureadahead ... 2014-04-01 16:44:19,897.897 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for man-db ... 2014-04-01 16:44:19,924.924 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for man-db ... 2014-04-01 16:44:21,308.308 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:21,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:21,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:21,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:21,775.775 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:21,777.777 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:21,941.941 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-common-dbg is not installed, so not removed 2014-04-01 16:44:21,942.942 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:21,951.951 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:22,020.020 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-common-dbg is not installed, so not removed 2014-04-01 16:44:22,021.021 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:22,031.031 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:22,199.199 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:22,200.200 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:22,280.280 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:22,282.282 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:22,404.404 INFO:teuthology.orchestra.run.out:[10.214.133.38]: The following packages will be REMOVED: 2014-04-01 16:44:22,405.405 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-fuse* ceph-fuse-dbg* 2014-04-01 16:44:22,485.485 INFO:teuthology.orchestra.run.out:[10.214.133.37]: The following packages will be REMOVED: 2014-04-01 16:44:22,486.486 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-fuse* ceph-fuse-dbg* 2014-04-01 16:44:22,612.612 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 2 to remove and 94 not upgraded. 2014-04-01 16:44:22,613.613 INFO:teuthology.orchestra.run.out:[10.214.133.38]: After this operation, 80.7 MB disk space will be freed. 2014-04-01 16:44:22,695.695 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 2 to remove and 94 not upgraded. 2014-04-01 16:44:22,696.696 INFO:teuthology.orchestra.run.out:[10.214.133.37]: After this operation, 80.7 MB disk space will be freed. 2014-04-01 16:44:22,755.755 INFO:teuthology.orchestra.run.out:[10.214.133.38]: (Reading database ... 80733 files and directories currently installed.) 2014-04-01 16:44:22,758.758 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-fuse-dbg ... 2014-04-01 16:44:22,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: (Reading database ... 80728 files and directories currently installed.) 2014-04-01 16:44:22,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-fuse-dbg ... 2014-04-01 16:44:23,174.174 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing ceph-fuse ... 2014-04-01 16:44:23,254.254 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing ceph-fuse ... 2014-04-01 16:44:23,653.653 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for man-db ... 2014-04-01 16:44:23,736.736 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for man-db ... 2014-04-01 16:44:25,111.111 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:25,188.188 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:25,485.485 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:25,488.488 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:25,563.563 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:25,565.565 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:25,752.752 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-fuse-dbg is not installed, so not removed 2014-04-01 16:44:25,753.753 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:25,767.767 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:25,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-fuse-dbg is not installed, so not removed 2014-04-01 16:44:25,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:25,842.842 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:26,128.128 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:26,130.130 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:26,219.219 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:26,221.221 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:26,377.377 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-test is not installed, so not removed 2014-04-01 16:44:26,378.378 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:26,391.391 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:26,480.480 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-test is not installed, so not removed 2014-04-01 16:44:26,481.481 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:26,491.491 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:26,723.723 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:26,724.724 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:26,740.740 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:26,741.741 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:26,921.921 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package ceph-test-dbg is not installed, so not removed 2014-04-01 16:44:26,922.922 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:26,933.933 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:26,953.953 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package ceph-test-dbg is not installed, so not removed 2014-04-01 16:44:26,953.953 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:26,962.962 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:27,211.211 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:27,212.212 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:27,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:27,272.272 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:27,395.395 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package radosgw is not installed, so not removed 2014-04-01 16:44:27,396.396 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:27,407.407 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:27,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package radosgw is not installed, so not removed 2014-04-01 16:44:27,523.523 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:27,523.523 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:27,752.752 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:27,754.754 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:27,766.766 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:27,768.768 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:27,948.948 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package radosgw-dbg is not installed, so not removed 2014-04-01 16:44:27,949.949 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:27,962.962 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:27,979.979 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package radosgw-dbg is not installed, so not removed 2014-04-01 16:44:27,980.980 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:27,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:28,238.238 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:28,240.240 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:28,307.307 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:28,310.310 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:28,430.430 INFO:teuthology.orchestra.run.out:[10.214.133.38]: The following packages will be REMOVED: 2014-04-01 16:44:28,431.431 INFO:teuthology.orchestra.run.out:[10.214.133.38]: python-ceph* 2014-04-01 16:44:28,540.540 INFO:teuthology.orchestra.run.out:[10.214.133.37]: The following packages will be REMOVED: 2014-04-01 16:44:28,541.541 INFO:teuthology.orchestra.run.out:[10.214.133.37]: python-ceph* 2014-04-01 16:44:28,638.638 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 1 to remove and 94 not upgraded. 2014-04-01 16:44:28,639.639 INFO:teuthology.orchestra.run.out:[10.214.133.38]: After this operation, 171 kB disk space will be freed. 2014-04-01 16:44:28,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 1 to remove and 94 not upgraded. 2014-04-01 16:44:28,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.37]: After this operation, 171 kB disk space will be freed. 2014-04-01 16:44:28,778.778 INFO:teuthology.orchestra.run.out:[10.214.133.38]: (Reading database ... 80722 files and directories currently installed.) 2014-04-01 16:44:28,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing python-ceph ... 2014-04-01 16:44:28,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: (Reading database ... 80717 files and directories currently installed.) 2014-04-01 16:44:28,792.792 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing python-ceph ... 2014-04-01 16:44:30,238.238 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:30,259.259 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:30,614.614 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:30,615.615 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:30,621.621 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:30,622.622 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:30,907.907 INFO:teuthology.orchestra.run.out:[10.214.133.37]: The following packages will be REMOVED: 2014-04-01 16:44:30,907.907 INFO:teuthology.orchestra.run.out:[10.214.133.37]: libcephfs1* libcephfs1-dbg* 2014-04-01 16:44:30,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: The following packages will be REMOVED: 2014-04-01 16:44:30,917.917 INFO:teuthology.orchestra.run.out:[10.214.133.38]: libcephfs1* libcephfs1-dbg* 2014-04-01 16:44:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 2 to remove and 94 not upgraded. 2014-04-01 16:44:31,062.062 INFO:teuthology.orchestra.run.out:[10.214.133.38]: After this operation, 90.8 MB disk space will be freed. 2014-04-01 16:44:31,085.085 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 2 to remove and 94 not upgraded. 2014-04-01 16:44:31,085.085 INFO:teuthology.orchestra.run.out:[10.214.133.37]: After this operation, 90.8 MB disk space will be freed. 2014-04-01 16:44:31,183.183 INFO:teuthology.orchestra.run.out:[10.214.133.38]: (Reading database ... 80709 files and directories currently installed.) 2014-04-01 16:44:31,186.186 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing libcephfs1-dbg ... 2014-04-01 16:44:31,187.187 INFO:teuthology.orchestra.run.out:[10.214.133.37]: (Reading database ... 80704 files and directories currently installed.) 2014-04-01 16:44:31,187.187 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing libcephfs1-dbg ... 2014-04-01 16:44:31,595.595 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Removing libcephfs1 ... 2014-04-01 16:44:31,631.631 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Removing libcephfs1 ... 2014-04-01 16:44:32,752.752 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Purging configuration files for libcephfs1 ... 2014-04-01 16:44:32,755.755 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Purging configuration files for libcephfs1 ... 2014-04-01 16:44:32,755.755 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Processing triggers for libc-bin ... 2014-04-01 16:44:32,756.756 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ldconfig deferred processing now taking place 2014-04-01 16:44:32,756.756 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Processing triggers for libc-bin ... 2014-04-01 16:44:32,756.756 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ldconfig deferred processing now taking place 2014-04-01 16:44:33,258.258 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading package lists... 2014-04-01 16:44:33,353.353 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading package lists... 2014-04-01 16:44:33,632.632 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Building dependency tree... 2014-04-01 16:44:33,638.638 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Reading state information... 2014-04-01 16:44:33,726.726 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Building dependency tree... 2014-04-01 16:44:33,728.728 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Reading state information... 2014-04-01 16:44:33,901.901 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Package libcephfs1-dbg is not installed, so not removed 2014-04-01 16:44:33,902.902 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:33,904.904 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 2014-04-01 16:44:33,989.989 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Package libcephfs1-dbg is not installed, so not removed 2014-04-01 16:44:33,990.990 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2014-04-01 16:44:33,991.991 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 2014-04-01 16:44:34,399.399 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove' 2014-04-01 16:44:34,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove' 2014-04-01 16:44:35,483.483 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:35,556.556 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:35,556.556 INFO:teuthology.task.install:Removing ceph sources lists 2014-04-01 16:44:35,557.557 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 16:44:36,062.062 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 16:44:36,062.062 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 16:44:36,064.064 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo rm -f /etc/apt/sources.list.d/ceph.list && sudo apt-get update || true' 2014-04-01 16:44:36,444.444 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 16:44:36,445.445 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo rm -f /etc/apt/sources.list.d/ceph.list && sudo apt-get update || true' 2014-04-01 16:44:40,807.807 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:40,955.955 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:40,956.956 INFO:teuthology.task.install:Purging /var/lib/ceph on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 16:44:40,957.957 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo rm -rf --one-file-system -- /var/lib/ceph || true ; test -d /var/lib/ceph && sudo find /var/lib/ceph -mindepth 1 -maxdepth 2 -type d -exec umount '{}' ';' ; sudo rm -rf --one-file-system -- /var/lib/ceph" 2014-04-01 16:44:40,958.958 INFO:teuthology.task.install:Purging /var/lib/ceph on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 16:44:40,958.958 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo rm -rf --one-file-system -- /var/lib/ceph || true ; test -d /var/lib/ceph && sudo find /var/lib/ceph -mindepth 1 -maxdepth 2 -type d -exec umount '{}' ';' ; sudo rm -rf --one-file-system -- /var/lib/ceph" 2014-04-01 16:44:41,319.319 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:41,447.447 DEBUG:teuthology.parallel:result is None 2014-04-01 16:44:41,448.448 DEBUG:teuthology.run_tasks:Unwinding manager clock.check 2014-04-01 16:44:41,448.448 INFO:teuthology.task.clock:Checking final clock skew... 2014-04-01 16:44:41,448.448 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 16:44:41,849.849 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: remote local st poll reach delay offset disp 2014-04-01 16:44:41,850.850 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: ======================================================================= 2014-04-01 16:44:41,850.850 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: *clock3.dreamhos 10.214.133.37 1 128 377 0.00034 -0.000013 0.09187 2014-04-01 16:44:41,957.957 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 16:44:42,330.330 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: remote local st poll reach delay offset disp 2014-04-01 16:44:42,331.331 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: ======================================================================= 2014-04-01 16:44:42,331.331 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: *clock3.dreamhos 10.214.133.38 1 128 377 0.00032 -0.000094 0.09421 2014-04-01 16:44:42,332.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2014-04-01 16:44:42,332.332 INFO:teuthology.task.internal:Duration was 4046.998991 seconds 2014-04-01 16:44:42,332.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2014-04-01 16:44:42,333.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2014-04-01 16:44:42,333.333 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2014-04-01 16:44:42,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2014-04-01 16:44:42,819.819 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2014-04-01 16:44:43,166.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2014-04-01 16:44:43,166.166 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump' 2014-04-01 16:44:43,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump' 2014-04-01 16:44:43,535.535 INFO:teuthology.orchestra.run.out:[10.214.133.38]: kernel.core_pattern = core 2014-04-01 16:44:43,849.849 INFO:teuthology.orchestra.run.out:[10.214.133.37]: kernel.core_pattern = core 2014-04-01 16:44:43,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "if test '!' -e /home/ubuntu/cephtest/archive/coredump ; then echo OK ; fi" 2014-04-01 16:44:44,499.499 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "if test '!' -e /home/ubuntu/cephtest/archive/coredump ; then echo OK ; fi" 2014-04-01 16:44:44,979.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2014-04-01 16:44:44,979.979 INFO:teuthology.task.internal:Removing archive directory... 2014-04-01 16:44:44,980.980 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'rm -rf -- /home/ubuntu/cephtest/archive' 2014-04-01 16:44:45,336.336 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'rm -rf -- /home/ubuntu/cephtest/archive' 2014-04-01 16:44:45,661.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2014-04-01 16:44:45,661.661 INFO:teuthology.task.internal:Tidying up after the test... 2014-04-01 16:44:45,662.662 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'rmdir -- /home/ubuntu/cephtest' 2014-04-01 16:44:45,982.982 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'rmdir -- /home/ubuntu/cephtest' 2014-04-01 16:44:46,338.338 INFO:teuthology.run:Summary data: {duration: 4046.9989910125732, flavor: basic, owner: loic@dachary.org, success: true} 2014-04-01 16:44:46,339.339 DEBUG:teuthology.report:No results_server set in /home/loic/.teuthology.yaml; not attempting to push results 2014-04-01 16:44:46,339.339 INFO:teuthology.run:pass